Beyond PII: Evaluating privacy in LLM and AI workflows

Learn how to move beyond simple PII removal and implement nuanced privacy evaluations for LLMs. This video explores mathematical metrics, building custom benchmarks, and using LLMs as judges or building out task-specific synthetic data to evaluate privacy of your AI workflows. Watch a practical code demonstration using synthetic data to stress-test your privacy controls. Related Links: Notebookfor this video: https://github.com/kjam/secure-and-pr... Eval FAQ: https://hamel.dev/blog/posts/evals-faq/ 4 major approaches to Evals: https://magazine.sebastianraschka.com... BERTScore: https://arxiv.org/abs/1904.09675 Arize Pheonix Evals: https://arize.com/docs/phoenix/evalua... Synthetic Data Eval Notebook: https://github.com/kjam/secure-and-pr... noyb: https://noyb.eu/en Confaide research (where the example data near end of video comes from): https://arxiv.org/abs/2310.17884 00:00 - Introduction to Privacy Evals 00:56 - General AI Security Metrics 01:30 - Accuracy and Precision Metrics 03:52 - Benchmark and Evaluation Datasets 05:42 - Building Custom Evaluations 07:32 - Jupyter Notebook Code Demo 08:16 - Testing PII Removal 10:20 - Generating Synthetic Data 13:14 - LLM as a Privacy Judge 16:40 - Privacy Reasoning Through Stories 19:06 - Comparing Different LLM Models 20:50 - Real-World Workflow Use Cases 23:18 - Conclusion and Next Steps