Building Agentic AI Projects for ML Image Classification: XGBoost vs. CNN.

Everyone is building “AI agents” by wrapping a large language model around their code. We tried that too. We wired Claude agent in as the orchestrator of a medical image classification pipeline, tested it, and then deleted it. What replaced it works better for this job: a deterministic agent that runs the whole pipeline on its own. It works out its own plan. Nobody tells it the order of steps; it derives the order from what each skill needs and produces. It picks the right tool. When cached features exist for the exact same data, it reuses them instead of recomputing: a 157-second run drops to 77 seconds. It handles trouble on its own. It stops on bad data, retries a failed save, and refuses to promote a model that fails its quality gates. It logs its reasoning. Every decision is written down with the rule behind it. And every run, by every route, produces a byte-identical model. This talk walks through that project end to end: a five-class lung CT-patch classifier (normal, benign granuloma, adenocarcinoma, squamous cell carcinoma, large cell carcinoma), built in Python with OpenCV, scikit-image and XGBoost. It reaches 96.67% test accuracy, 0.998 ROC-AUC and 0.958 MCC. Then we put it under pressure, with real numbers: Five ways to build the same pipeline: a plain script, an agent with skills, a LangGraph version with human approval, an LLM-orchestrated version and the final deterministic agent. Which ones were worth the code, and which were not? Deep learning vs. classic ML. A Keras CNN on identical images reached 93.17% after 33 minutes of training. XGBoost reached 96.67% in 10 seconds. Why the “old” method won. Can you skip labels? We tried six unsupervised clustering approaches (K-Means, hierarchical, DBSCAN, PCA, autoencoders, Gaussian mixtures). The best agreed with the diagnoses 87% of the time, but missed over a third of one cancer type. What clustering is actually good for in production. Real data hits back. On an open lung CT dataset, a…