Eval Consulting

Al Developer

Eval ConsultingContract
Remote
4 - 10 YearsMar 3rd, 2026
94 ViewsBe an Early Applicant
Required Skillset:
PythonAzureKafkaSemantic KernelPineconeWeaviateMilvussecurityaudit loggingAWSGCPgovernanceDevOpsSQSvector databasesServiceNowsearchLangChainrisk assessmentsfinancial servicesLangGraphprivacy principlesleast privilegecustom workflow enginesLlamalndexPIl handlingregulated domainsrisk controlscompliance audit readinessenterprise workflowsBPM/RPALLM-as-judgerubric scoringretrieval evalsoffline/online testingmessaging/eventingemail ingestion pipelinesdocument processingMRM concernsmodel cardsvalidation processesback-end services/APIsOpenSearch/Elasticmodel evaluation approaches

Job Description

  • 4+ years of software engineering experience or equivalent with strong CS fundamentals
  • Hands-on experience building with LLMs and modern Al app stack (agents, RAG, tool/function calling).
  • Strong proficiency in Python and building back-end services/APls.
  • Experience with at least one: LangChain/ LangGraph, Llamalndex, Semantic Kernel or equivalent frameworks.
  • Experience with vector databases and search (e.g., Pinecone, Weaviate, Milvus, OpenSearch/Elastic, )
  • Experience deploying services in cloud environments (AWS/Azure/GP) with basic DevOps practices
  • Strong understanding of security and privacy principles (PIl handling, least privilege, audit logging)
  • Preferred Qualifications
  • Experience in financial services or other regulated domains (risk controls, compliance audit readiness)
  • Experience integrating with enterprise workflows (e.g., ServiceNow, Custom workflow engines,
    BPM/RPA)
  • Familiarity with model evaluation approaches (LLM-as-judge, rubric scoring, retrieval evals, offline/online testing)
  • Experience with messaging/eventing (Kafka/SQS), email ingestion pipelines, and document processing
  • Exposure to MRM concerns and governance (model cards, risk assessments, validation processes)

Preferred, but not required:

  • Experience in financial services or regulated domains (risk controls, compliance).
  • Familiarity with enterprise workflow integrations (e.g., ServiceNow, RPA, BPM).
  • Knowledge of model evaluation techniques and testing approaches.
  • Exposure to messaging/eventing systems (Kafka/SQS), document processing, and ingestion pipelines.
  • Understanding of MRM governance, model cards, risk assessments, and validation processes.

 

Similar Jobs

Net Developer

Remote

Feb 27th, 2026

Senior Developer

Remote

Feb 27th, 2026

Sr Developer

Remote

Feb 26th, 2026

Senior Developer

Connecticut

Feb 24th, 2026

Al Developer

North Carolina

Feb 11th, 2026