
Al Developer
Eval ConsultingContract
Required Skillset:
PythonAzureKafkaSemantic KernelPineconeWeaviateMilvussecurityaudit loggingAWSGCPgovernanceDevOpsSQSvector databasesServiceNowsearchLangChainrisk assessmentsfinancial servicesLangGraphprivacy principlesleast privilegecustom workflow enginesLlamalndexPIl handlingregulated domainsrisk controlscompliance audit readinessenterprise workflowsBPM/RPALLM-as-judgerubric scoringretrieval evalsoffline/online testingmessaging/eventingemail ingestion pipelinesdocument processingMRM concernsmodel cardsvalidation processesback-end services/APIsOpenSearch/Elasticmodel evaluation approaches
Job Description
- 4+ years of software engineering experience or equivalent with strong CS fundamentals
- Hands-on experience building with LLMs and modern Al app stack (agents, RAG, tool/function calling).
- Strong proficiency in Python and building back-end services/APls.
- Experience with at least one: LangChain/ LangGraph, Llamalndex, Semantic Kernel or equivalent frameworks.
- Experience with vector databases and search (e.g., Pinecone, Weaviate, Milvus, OpenSearch/Elastic, )
- Experience deploying services in cloud environments (AWS/Azure/GP) with basic DevOps practices
- Strong understanding of security and privacy principles (PIl handling, least privilege, audit logging)
- Preferred Qualifications
- Experience in financial services or other regulated domains (risk controls, compliance audit readiness)
- Experience integrating with enterprise workflows (e.g., ServiceNow, Custom workflow engines,
BPM/RPA) - Familiarity with model evaluation approaches (LLM-as-judge, rubric scoring, retrieval evals, offline/online testing)
- Experience with messaging/eventing (Kafka/SQS), email ingestion pipelines, and document processing
- Exposure to MRM concerns and governance (model cards, risk assessments, validation processes)
Preferred, but not required:
- Experience in financial services or regulated domains (risk controls, compliance).
- Familiarity with enterprise workflow integrations (e.g., ServiceNow, RPA, BPM).
- Knowledge of model evaluation techniques and testing approaches.
- Exposure to messaging/eventing systems (Kafka/SQS), document processing, and ingestion pipelines.
- Understanding of MRM governance, model cards, risk assessments, and validation processes.
Similar Jobs
Net Developer
Remote
Feb 27th, 2026
Senior Developer
Remote
Feb 27th, 2026
Sr Developer
Remote
Feb 26th, 2026
Senior Developer
Connecticut
Feb 24th, 2026
Al Developer
North Carolina
Feb 11th, 2026