Announcing GA launch of Kafka Trigger extension on Azure Functions

Announcing GA launch of Kafka Trigger extension on Azure Functions

Published Jun 23 2022 06:04 AM 862 Views

Skip to footer content

With the power of serverless event driven compute platform and real time stream processing, application development can evolve from traditional monolith apps to programming models whose flows are based on internal or external events and not just on its structure.

Serverless compute allows you to run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. Serverless Stream processing is increasingly becoming a vital part of complex distributed systems. An event streaming platform like Kafka is capable of handling trillions of events, manipulates data as it arrives, supports publishing of data to any number of systems or real-time applications running in distributed systems.

Serverless architectures are well-suited for stream processing workloads which are often event-driven and have spiky or variable compute requirements. Azure functions in combination with Kafka are powerful and easy to use services which will enable you to build real-time event driven applications running on distributed systems.

We are happy to announce the GA release of Kafka trigger which enables you to invoke functions in response to messages in Kafka topics and lets you write values/messages out to Kafka topics using an output binding.  Just focus on your Azure Function’s logic without worrying about the event-sourcing pipeline or maintaining infra to host the extension.  This extension is supported when hosting functions in the Premium plan enabling it to elastically scale and trigger on Kafka messages.  

The new features that are made available in GA with the Kafka extension are as follows:

Now let's take a look at the scenario sample show casing the power of Azure Functions Kafka trigger extension integrating with Kafka brokers hosted on Confluent cloud.

The goal of this use-case is to process the wallet transaction and notify users via email/SMS/in-app notifications. This is a fictitious example to showcase the Kafka trigger and output binding usage with Avro and header data. The code can be further customized and extended as per your requirements.

The wallet producer helps generate wallet transactions in the Avro serialized format to Kafka topic- wallet_event.

The flow of the application is captured in this diagram:

Check out the step-by-step instructions of this sample available here Azure Functions Kafka Trigger Java Sample - Code Samples | Microsoft Docs 

Summary

In this blog you have seen how to use the latest additions to Azure functions Kafka extension with the help of a use-case scenario showcasing the integration between Azure functions and Kafka (running on Confluent cloud). Event driven serverless compute services are gaining popularity while building cloud-native microservices apps. Serverless architectures are well-suited for stream processing workloads which are often event-driven and have spiky or variable compute requirements thus enabling you to focus on achieving a specific business-related goal, without any concerns for how much computing power is needed, or how you’ll handle availability.

Want to learn more?

Please refer to Apache Kafka bindings for Azure Functions | Microsoft Docs for detail documentation, samples on the Azure function supported languages and more!

References

Apache Kafka bindings for Azure Functions | Microsoft Docs

 If you encounter any issues or would like to provide feedback on Kafka trigger extension, please report them to our GitHub repository- Issues · Azure/azure-functions-kafka-extension (github.com)

This extension is being developed in the open-source community. Please contribute, try out and post any issues on the Azure Functions Kafka extension GitHub repo

Keep in touch with us on Twitter via @AzureFunctions.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

%3CLINGO-SUB%20id%3D%22lingo-sub-3499015%22%20slang%3D%22en-US%22%3EAnnouncing%20GA%20launch%20of%20Kafka%20Trigger%20extension%20on%20Azure%20Functions%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3499015%22%20slang%3D%22en-US%22%3E%3CP%3EWith%20the%20power%20of%20serverless%20event%20driven%20compute%20platform%20and%20real%20time%20stream%20processing%2C%20application%20development%20can%20evolve%20from%20traditional%20monolith%20apps%20to%20programming%20models%20whose%20flows%20are%20based%20on%20internal%20or%20external%20events%20and%20not%20just%20on%20its%20structure.%3C%2FP%3E%0A%3CP%3EServerless%20compute%20allows%20you%20to%20run%20code%20without%20provisioning%20or%20managing%20servers%2C%20creating%20workload-aware%20cluster%20scaling%20logic%2C%20maintaining%20event%20integrations%2C%20or%20managing%20runtimes.%20Serverless%20Stream%20processing%20is%20increasingly%20becoming%20a%20vital%20part%20of%20complex%20distributed%20systems.%20An%20event%20streaming%20platform%20like%20Kafka%20is%20capable%20of%20handling%20trillions%20of%20events%2C%20manipulates%20data%20as%20it%20arrives%2C%20supports%20publishing%20of%20data%20to%20any%20number%20of%20systems%20or%20real-time%20applications%20running%20in%20distributed%20systems.%3C%2FP%3E%0A%3CP%3EServerless%20architectures%20are%20well-suited%20for%20stream%20processing%20workloads%20which%20are%20often%20event-driven%20and%20have%20spiky%20or%20variable%20compute%20requirements.%20Azure%20functions%20in%20combination%20with%20Kafka%20are%20powerful%20and%20easy%20to%20use%20services%20which%20will%20enable%20you%20to%20build%20real-time%20event%20driven%20applications%20running%20on%20distributed%20systems.%3C%2FP%3E%0A%3CP%3EWe%20are%20happy%20to%20announce%20the%20GA%20release%20of%20Kafka%20trigger%20which%20enables%20you%20to%20invoke%20functions%20in%20response%20to%20messages%20in%20Kafka%20topics%20and%20lets%20you%20write%20values%2Fmessages%20out%20to%20Kafka%26nbsp%3Btopics%20using%20an%20output%20binding.%20%26nbsp%3BJust%20focus%20on%20your%20Azure%20Function%E2%80%99s%20logic%20without%20worrying%20about%20the%20event-sourcing%20pipeline%20or%20maintaining%20infra%20to%20host%20the%20extension.%20%26nbsp%3BThis%20extension%20is%20supported%20when%20hosting%20functions%20in%20the%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdocs.microsoft.com%252Fen-us%252Fazure%252Fazure-functions%252Ffunctions-scale%2523premium-plan%26amp%3Bdata%3D05%257C01%257CRamya.Oruganti%2540microsoft.com%257C40ae74348c114eed07df08da370281a4%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637882781148981737%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C3000%257C%257C%257C%26amp%3Bsdata%3DeCulLNQ2iszb7sDmuL4fzT4UcCmOaOwezuAMQD6tz8A%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EPremium%20plan%3C%2FA%3E%26nbsp%3Benabling%20it%20to%20elastically%20scale%20and%20trigger%20on%20Kafka%20messages.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-functions%2Ffunctions-bindings-kafka%3Ftabs%3Din-process%252Cportal%26amp%3Bpivots%3Dprogramming-language-csharp%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Enew%20features%3C%2FA%3E%20that%20are%20made%20available%20in%20GA%20with%20the%20Kafka%20extension%20are%20as%20follows%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EBoth%20trigger%20and%20output%20binding%20now%20support%20Kafka%20Headers%20for%20all%20languages%20that%20Azure%20Functions%20support%20(C%23%2C%20Java%2C%20Python%2C%20PowerShell%2C%20Node).%20The%20header%20data%20can%20be%20passed%20on%20to%20the%20function%E2%80%99s%20trigger%20that%20gets%20executed%20and%20can%20be%20further%20processed.%20As%20per%20your%20requirements%20the%20header%20data%20can%20flow%20to%20Kafka%20topic%20of%20your%20choice%20using%20the%20output%20binding.%3C%2FLI%3E%0A%3CLI%3EAvro%20Deserialization%20for%20Generic%20Records%20is%20now%20supported%20for%20all%20languages%20that%20Azure%20functions%20support%20(C%23%2C%20Java%2C%20Python%2C%20PowerShell%2C%20Node).%3C%2FLI%3E%0A%3CLI%3EKafka%20trigger%20function%20apps%20in%20Java%20can%20be%20created%20through%20the%20templates%20using%20maven.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3ENow%20let's%20take%20a%20look%20at%20the%20scenario%20sample%20show%20casing%20the%20power%20of%20Azure%20Functions%20Kafka%20trigger%20extension%20integrating%20with%20Kafka%20brokers%20hosted%20on%20Confluent%20cloud.%3C%2FP%3E%0A%3CP%3EThe%20goal%20of%20this%20use-case%20is%20to%20process%20the%20wallet%20transaction%20and%20notify%20users%20via%20email%2FSMS%2Fin-app%20notifications.%20This%20is%20a%20fictitious%20example%20to%20showcase%20the%20Kafka%20trigger%20and%20output%20binding%20usage%20with%20Avro%20and%20header%20data.%20The%20code%20can%20be%20further%20customized%20and%20extended%20as%20per%20your%20requirements.%3C%2FP%3E%0A%3CP%3EThe%20wallet%20producer%20helps%20generate%20wallet%20transactions%20in%20the%20Avro%20serialized%20format%20to%20Kafka%20topic-%20wallet_event.%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EOnce%20the%20message%20arrives%20at%20a%20Kafka%20topic%20a%20function%20app%20is%20triggered%20which%20de-serializes%20the%20transaction%20data%20and%20initiates%20header%20data%20which%20is%20used%20to%20send%20notifications%20accordingly.%3C%2FLI%3E%0A%3CLI%3EThen%20a%20new%20message%20is%20constructed%20with%20header%20data%20and%20this%20message%20is%20then%20sent%20to%20notification_event_topic%20using%20the%20Kafka%20output%20binding%3C%2FLI%3E%0A%3CLI%3EThe%20Notification%20listener%20will%20be%20invoked%20once%20there%20is%20a%20new%20message%20in%20the%20kafka%20topic%20-%20notification_event_topic%2C%20then%20the%20respective%20notification%20events%20are%20passed%20on%20to%20the%20email%2C%20SMS%20and%20in-app%20topics%20using%20the%20kafka%20output%20bindings%20which%20are%20picked%20by%20the%20corresponding%20event%20function%20apps.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EThe%20flow%20of%20the%20application%20is%20captured%20in%20this%20diagram%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22arch_diag.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3Cspan%20class%3D%22lia-inline-image-display-wrapper%22%20image-alt%3D%22arch_diag.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3Cimg%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F383003i75178873BB2ACF18%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22arch_diag.png%22%20alt%3D%22arch_diag.png%22%20%2F%3E%3C%2Fspan%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECheck%20out%20the%20step-by-step%20instructions%20of%20this%20sample%20available%20here%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsamples%2Fazure%2Fazure-functions-kafka-extension%2Fazure-functions-kafka-trigger-java-sample%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20Functions%20Kafka%20Trigger%20Java%20Sample%20-%20Code%20Samples%20%7C%20Microsoft%20Docs%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ESummary%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EIn%20this%20blog%20you%20have%20seen%20how%20to%20use%20the%20latest%20additions%20to%20Azure%20functions%20Kafka%20extension%20with%20the%20help%20of%20a%20use-case%20scenario%20showcasing%20the%20integration%20between%20Azure%20functions%20and%20Kafka%20(running%20on%20Confluent%20cloud).%20Event%20driven%20serverless%20compute%20services%20are%20gaining%20popularity%20while%20building%20cloud-native%20microservices%20apps.%20Serverless%20architectures%20are%20well-suited%20for%20stream%20processing%20workloads%20which%20are%20often%20event-driven%20and%20have%20spiky%20or%20variable%20compute%20requirements%20thus%20enabling%20you%20to%20focus%20on%20achieving%20a%20specific%20business-related%20goal%2C%20without%20any%20concerns%20for%20how%20much%20computing%20power%20is%20needed%2C%20or%20how%20you%E2%80%99ll%20handle%20availability.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EWant%20to%20learn%20more%3F%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EPlease%20refer%20to%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-functions%2Ffunctions-bindings-kafka%3Ftabs%3Din-process%252Cportal%26amp%3Bpivots%3Dprogramming-language-csharp%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EApache%20Kafka%20bindings%20for%20Azure%20Functions%20%7C%20Microsoft%20Docs%3C%2FA%3E%20for%20detail%20documentation%2C%20samples%20on%20the%20Azure%20function%20supported%20languages%20and%20more!%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EReferences%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-functions%2Ffunctions-bindings-kafka%3Ftabs%3Din-process%252Cportal%26amp%3Bpivots%3Dprogramming-language-csharp%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EApache%20Kafka%20bindings%20for%20Azure%20Functions%20%7C%20Microsoft%20Docs%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3BIf%20you%20encounter%20any%20issues%20or%20would%20like%20to%20provide%20feedback%20on%20Kafka%20trigger%20extension%2C%20please%20report%20them%20to%20our%20GitHub%20repository-%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2Fazure-functions-kafka-extension%2Fissues%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EIssues%20%C2%B7%20Azure%2Fazure-functions-kafka-extension%20(github.com)%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EThis%20extension%20is%20being%20developed%20in%20the%20open-source%20community.%20Please%20contribute%2C%20try%20out%20and%20post%20any%20issues%20on%20the%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2Fazure-functions-kafka-extension%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20Functions%20Kafka%20extension%20GitHub%20repo%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EKeep%20in%20touch%20with%20us%20on%20Twitter%20via%20%40%3CA%20href%3D%22https%3A%2F%2Ftwitter.com%2FAzureFunctions%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EAzureFunctions%3C%2FA%3E.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-3499015%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Ramya_Oruganti_0-1655110755580.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3Cspan%20class%3D%22lia-inline-image-display-wrapper%22%20image-alt%3D%22Ramya_Oruganti_0-1655110755580.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3Cimg%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F379730i5E17AF73186160D0%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Ramya_Oruganti_0-1655110755580.png%22%20alt%3D%22Ramya_Oruganti_0-1655110755580.png%22%20%2F%3E%3C%2Fspan%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EAnnouncing%20the%20general%20availability%20of%20the%E2%80%AF%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fgithub.com%252FAzure%252Fazure-functions-kafka-extension%26amp%3Bdata%3D05%257C01%257CRamya.Oruganti%2540microsoft.com%257C40ae74348c114eed07df08da370281a4%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637882781148981737%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C3000%257C%257C%257C%26amp%3Bsdata%3DS6COJDE6XCRQT2TfKjhNDhmCJnmFRzdtTXlaEALAW3Y%253D%26amp%3Breserved%3D0%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%20target%3D%22_blank%22%3EKafka%20Extension%3C%2FA%3E%E2%80%AFfor%20Azure%20Functions.%20This%20extension%20will%20enable%20you%20to%20detect%20and%20%26nbsp%3Brespond%20%26nbsp%3Bto%20real%20time%20messages%20streaming%20into%20Kafka%20Topics%2C%20or%20write%20to%20a%20Kafka%20Topic%20through%20the%20output%20binding.%20%26nbsp%3BJust%20focus%20on%20your%20Azure%20Function%E2%80%99s%20logic%20without%20worrying%20about%20the%20event-sourcing%20pipeline%20or%20maintaining%20infra%20to%20host%20the%20extension.%20%26nbsp%3BKafka%20extension%20now%20supports%20Kafka%20Headers%20on%20trigger%20%26amp%3B%20output%20binding%20and%20Avro%20Deserialization%20for%20Generic%20Records%20for%20all%20languages%20now.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-3499015%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Functions%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EJava%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EServerless%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E