185: Workflow Orchestrators

Intro topic: Asymmetric ReturnsNews/Links:NanoChat by Andrej Karpathyhttps://github.com/karpathy/nanochatPydantic AIhttps://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/1000th Starlink this yearhttps://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/ChatGPT Apps SDKhttps://openai.com/index/introducing-apps-in-chatgpt/Book of the ShowPatrickThe Will of the Many by James Islingtonhttps://amzn.to/43IfU8QJasonInterview with DHH (Founder of Ruby on Rails)https://www.youtube.com/watch?v=vagyIcmIGOQPatreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrickFactoriohttps://www.factorio.com/ Jasonnip.io Topic: Workflow OrchestratorsWhyBatch jobs (embarrassingly parallel)Long-running tasks (e.g. transcoding video)Checkpointing/resumingHowMessage QueuesContainerizationWorker Pools & AutoscalingHistory & BackfillSteps to run workflows:Containerize the workflow definition and send to the cloudContainerize all the individual tasksSubmit job(s)ExamplesAirflowLegacy but dominantDagsterGreat UX for python developersTemporal: https://temporal.io/The new hotnessRayLow-level but very powerfulKubeflowDesigned for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
]]

Programming Throwdown

Patrick Wheeler and Jason Gauci

184: Asynchronous Programming

186: Becoming a Manager

Download Audio File

**Intro topic: Asymmetric Returns

**News/Links:

• NanoChat by Andrej Karpathy

• https://github.com/karpathy/nanochat

• Pydantic AI

• https://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/

• 1000th Starlink this year

• https://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/

• ChatGPT Apps SDK

• https://openai.com/index/introducing-apps-in-chatgpt/

Book of the Show

• Patrick

• The Will of the Many by James Islington

• https://amzn.to/43IfU8Q

• Jason

• Interview with DHH (Founder of Ruby on Rails)

• https://www.youtube.com/watch?v=vagyIcmIGOQ

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick

• Factorio

• https://www.factorio.com/

• Jason

• nip.io

**Topic: Workflow Orchestrators

• Why

• Batch jobs (embarrassingly parallel)
• Long-running tasks (e.g. transcoding video)
• Checkpointing/resuming

• How

• Message Queues
• Containerization
• Worker Pools & Autoscaling
• History & Backfill

• Steps to run workflows:

• Containerize the workflow definition and send to the cloud
• Containerize all the individual tasks
• Submit job(s)

• Examples

• Airflow

• Legacy but dominant

• Dagster

• Great UX for python developers

• Temporal: https://temporal.io/

• The new hotness

• Ray

• Low-level but very powerful

• Kubeflow

• Designed for ML workflows, integrated dashboard

Original source