Tutorial: Getting Started with Azure Management Libraries for Java

Six Things That Are Good To Know When Running Linux On Azure

Social Impact of Big and Open Data

Top 10 Openness Posts | August 2014

How to Use Open Source Hive with Azure HDInsight? Big Data management tools are becoming more essential and expected in business. A must. As we are consistently infororming you about the open source trend which has been sprouting in Microsoft in many areas, today we would like to bring to your attention some open source big data projects which are available in HDInsight and Azure. Shall we begin?  What is Hive? In the words of the project developers: “Hive is an open source volunteer project under the Apache Software Foundation. Previously it was a subproject of Apache Hadoop, but has now graduated to become a top-level project of its own.” Apache Hive provides a means of running MapReduce job through an SQL-like scripting language, called HiveQL. Hive is a data warehouse system for Hadoop, which enables data summarization, querying, and analysis of large volumes of data. More on documentation, community, contributions and development is to be found on the Hive site. Lets also not forget about the film from the beginning of this article  - a Hadoop Tutorial: Apache Hive from Hortonworks University.  And for all those of you beginning with HDInsight… A LOT of nicely organized information is presented on the main project website HDInsight: Hadoop in Azure - which is a 100% Apache Hadoop-based service in the cloud - where you will find information on the languages you can develop in, what types of data you can process, creating Hadoop clusters just within a few minutes, and much, much more. And with this tutorial you will feel as if somebody has taken you by the hand and helped you with getting started with Hadoop in HDInsight. But if you are not in the mood for searching & surfing, you can go to Channel 9 films in the Getting started with Windows Azure HDInsight Service section, where you will find 10 short videos (we mean really short, as the shortest is 1 minute, and the longest around 11 minutes). However, if you need more rock-solid knowledge, we recomend this free ebook Introducing Microsoft Azure HDInsight, which is intended to help database and business intelligence (BI) professionals, programmers, Hadoop administrators, researchers, technical architects, operations engineers, data analysts, and data scientists understand the core concepts of HDInsight and related technologies. The post also nicely sums up what you can find in the book, so that you can decide whether it will be useful for you before you actually start downloading it. But enough of general introductions. It’s time to get down to business. HDInsight + Hive  In this team blog post (rich in screenshots, links and code) from the Big Data Support team at Microsoft: Get Started with Hive on HDInsight you will learn about Hive implementation on HDInsight, provisioning a HDInsight cluster, executing a Hive Query on a newly provisioned cluster and upgrading the cluster. Fig 1. Hive Architecture on HDInsight from the article: “Get Started with Hive on HDInsight" from the the Big Data Support team at Microsoft. Another great article “Use Hive with Hadoop in HDInsight" will take you through: The Hive usage case, Uploading data for Hive tables, Running Hive queries using PowerShell, Using Tez for improved performance And finally, on this Apache Hive Confluence site you will learn all there is to know on getting started and the language manual. We also recommend that you follow the Curah! Use Hive with HDInsight website curated by Jonathan Gao, a technical writer at Microsoft specializing in HDInsight, which will keep you updated about any news in this subject. But while we are on the subject of HDInsight, we would like to let you know about one more thing.  Azure HDInsight + HBase This is quite fresh news, so not everybody might know that Azure HDInsight has made HBase (NoSQL database) a generally available feature. What is Apache HBase anyway? Apache HBase is the Hadoop database, a distributed, scalable, big data store. You can use it when you need random, realtime read/write access to your Big Data. As we can read on the project’s website: “This project’s goal is the hosting of very large tables — billions of rows X millions of columns.” How can you use HBase in practice? Here are some examples from Oliver Chiu: Internet of Things – HBase could be used as the storage for the millions of real time events coming from devices, sensors, equipment/machinery and social media. Hadoop with HDInsight can then perform batch analysis on the data that was stored in Azure Blobs. Web Logs – store and index web logs and clickstream data using HBase. Hadoop with HDInsight can then do batch analysis on this data. Social Sentiment – Use HBase to write and store data from the social sentiment fire hose (example: Twitter). This article will guide you to more information about HBase on Azure, like HDInsight documentation and getting started guides.   If you are really into a combination of Hive and HDInsight, remember to follow the Use Hive with HDInsight website, but please don’t forget about us, because we will be serving more on open source and Microsoft, and we want our community to get the hottest and most important information in this field! - The Microsoft Openness At CEE team

Linux Integration Services for Hyper-V   This 40 minute Edge Show interview with Abhishek Gupta, Program Manager for Linux and BSD Integration Services at Microsoft, covers new release functionality, availability for various distributions, as well as previewing the new Generation 2 VM capabilities for Linux Guests. In this interview Abhishek shares some great reference information, as well as you will get to see demos showing off Linux Integration Services 3.5.  He says: "I am one of the guys who is entrusted with ensuring that all Open Source operating systems run extremely well on Microsoft’s virtualization technologies such as Hyper-V and Microsoft Azure, and today I will be talking about what infrastructure is available to run Linux and FreeBSD on Hyper-V." The relationship of Microsoft and Linux Just as a reminder, Linux runs as a first class citizen on Windows Server Hyper-V and is supported on Azure Virtual Machines (CentOS, openSUSE, SUSE, Ubuntu). Here you can find a variety of different resources on Linux and Microsoft. If you would like to take your dose of knowledge in the form of a film – you can watch Load Balancing Linux Services On Azure or Embracing Linux Devices on Azure via Service Bus and Web Sites, but if you would prefer a tutorial, then perhaps you might be interested in Building and Modifying A Linux Kernel With Visual Studio in 20 Steps. And just in case you haven’t heard, Microsoft Joins the Linux Foundation AllSeen Alliance, which is a big step in Microsoft’s support of Open Source projects.  More related news Here are also a few links suggested by the Edge Team which you might find useful:  Best Practices for Operating your Hybrid Cloud from Brad Anderson at “In the cloud” blog. Mixed Tape for Best Blog Posts for 2013 from the Ask PFE blog. Linux Integration Services v 3.5 available for download blog post from Abhishek Gupta.  As the Edge Team says in the film: "If you are a person that is a Linux type of folk that likes to run that on systems centers / Hyper-V - you definitely want to check this out!” We hope you find information shared in this interview useful, and we promise to have a lot more Microsoft + Linux news for you soon!  - The Openness At CEE Team

What Convinced These Companies to Choose Microsoft Azure?

Open Source Kaltura Community Edition Video Platform Available on Azure   Video is now a part of everyday experience – it plays an important role in learning, working or shopping, but we also publish and share more and more films in our private conversations on Social Media and the Internet. Now you can share short videos on Vine and Instragram, and Facebook has also recently made changes to improve video featuring. Today – anybody can be a filmmaker and practically any mobile device allows you to make a decent quality film and share it on the internet in real-time. Devices like GoPro (is there actually anything similar to it?) give totally new possibilities to filmmaking, you can even say they have created a whole new category of sport & activity filming. Microsoft being aware of this trend and most of all of user needs, has just recently made managing video assets and complex, media-heavy websites on Microsoft Azure much easier.  As the MS Open Tech team reports, “The open source Kaltura Community Edition is now available as a Linux-based image on VM Depot or as a custom installer available on GitHUB, both provided by Akvelon.” Do you know Kaltura? As we can read on the company’s website: "Kaltura is the world’s first Open Source Online Video Platform, providing both enterprise level commercial software and services, fully supported and maintained by Kaltura, as well as 100% free open-source community supported solutions, for video publishing, management, syndication and monetization.” The Kaltura platform includes industry leading media management applications as well as a framework for developing custom applications. Let’s take a look at Kaltura in numbers: 300,000 websites using Kaltura 50,000 developer community members 8 years on the market Kaltura is a leader in the area of ‘open video’ and also a founding member of the Open Video Alliance, a coalition of organizations dedicated to fostering open standards for online video. Kaltura Features Kaltura is also rich in features, offering 19 different options, amongst which you will find: Mobile & HTML5 Video Player Webcasting Live Streaming Web Conferencing Video Analytics Kaltura also allows users to develop their own solutions – either  via open API, or via the Kaltura Exchange. This open source platform powers video for large media companies, enterprises and educational instituations amongst whose customers you will find HBO, Zappos or Philips. Kaltura + Azure Integration Kaltura alone offers great possibilities, but paired with Microsoft Azure Media Services, Kaltura on Azure makes a great solution for complex media scenarios. Integration between Kaltura Server and Microsoft Azure enables a number of new possibilities. Examples: a video or audio file that has been uploaded to KMC (Kaltura Management Console) can be sent to Microsoft Azure Media Services for encoding, Kaltura players can also pull media content directly from Microsoft Azure’s CDN and storage. Ok, so how do I set up? The Microsoft Open Technologies team explain in their article how to set up Kaltura in Azure Media Services using either a fresh Linux on your VM or a prepackaged VM available on VM Depot containing the Kaltura services. Some of you are already familiar with flexible and accessible media services, others don’t have such experience yet, but both groups – if you ever need a solid video platform – try out the combination of Kaltura + Azure services! - The Openness At CEE Team 

Microsoft Joins the Linux Foundation AllSeen Alliance It is clear to see, that the Internet of Things is gaining speed and importance compared year to year. Microsoft is seriously committed to creating solutions that will work in the IoT world, and it is not pursuing this goal alone. Apart from creating the Windows Developer Program for IoT and being involved in IoT projects like Great River Medical Centre or the London Underground, the company has also joined the AllSeen Alliance created by the Linux Foundation. AllSeen Alliance – what is it about?  In an article titled: Microsoft backs open source for the Internet of Things ComputerWorld.com writes: “The Linux Foundation, a nonprofit consortium that promotes Linux adoption, late last year announced the creation of the AllSeen Alliance to promote an open source code framework to enable devices to discover one another and then connect and interact.  […]The AllSeen Alliance is an effort to standardize device communications. The code that it champions, called AllJoyn, was initially developed by Qualcomm but was subsequently made open source.” Andy Castonguay, an analyst at Machina Research, a research firm focusing on machine-to-machine (M2M) communications and the Internet of Things, adds: "AllSeen is a well-founded effort to create an open lingua franca among device manufacturers and developers that will benefit from the growing diversity of device brands in the global market." Microsoft’s role in the IoT revolution In the same article Andy Castonguay says: ”The addition of Microsoft to the alliance is certainly a boost to the growing array of companies joining the AllSeen effort”. PC World also notices that: “Microsoft is a good addition for the AllSeen Alliance.” You may be wondering however, why is Microsoft joining the alliance, anyway? As it is framed on the Windows Embedded blog: ”[…]on the AllSeen Alliance, our team is helping to create a universal software framework for IoT, to ease and speed innovation”.  Microsoft is proud to be in the company of 50 other members like Sharp, Panasonic, LG, Cisco and HTC, just to name w few. Azure and the Internet of Things Within Microsoft there has been a rapid pace of development of Azure-based services. Technologies like Microsoft Azure Intelligent Systems Service, Microsoft Azure Machine Learning (Azure ML) and powerful new data visualization capabilities are tools, which will help users improve productivity, profitability and offer new possibilities in the mobile-first, cloud-first world that we are living in. Example?  "We need a flexible solution" CGI is the world’s fifth-largest independent IT and business process services firm, who has developed a solution that securely connects ThyssenKrupp’s “things” — the thousands of sensors and systems in its elevators. CGI relies on the flexibility of Microsoft technologies, as John Hicklin, the principal Internet of Things consultant for CGI says: “Azure is quite inclusive, in terms of things that it’s able to host, and the Intelligent Systems Service is inclusive in terms of devices it can connect with. If you put those two things together, you have a very rich cloud environment that is enterprise-ready, and also the ability to deal with the real world and with different types of devices and applications.” You will find detailed information on this case study here.  Don’t mind the estimations, it’s going to be….big!  We don’t want to bore you with numbers, however we find it quite interesting to see how this whole IoT phenomenon is perceived by different analytics. Windows Embedded General Manager Kevin Dallas in the 5:10 minute film at the beginning of our article explores just how big is this opportunity: 50 bln connected devices according to Cisco and Ericsson 75 bln connected devices by Morgan Stanley 212 bln according to the IDC by 2020. We also found an estimation tweeted by Bosch stating that there will be 14 billion connected devices by 2022. image source: @BoschGlobal Microsoft has also made a strong statement about this estimation….. which you just have to see in the film :) Well, we will just have to wait and see whose predictions turned out to be the most accurate. And while Microsoft is working towards establishing interoperability standards and common architectures to connect smart devices, machines, people, processes and data, we just can’t wait what you – the community, developers, and users - are going to do with it. - The Microsoft Openness At CEE team

Packer & OpenNebula on Microsoft Azure

Enhancing Creative Development with Kinect Common Bridge We especially enjoy sharing news about projects, which stir the imagination and open new creative possibilities. And it seems that KCB has given designers a new tool for creating new user experiences. And what is so special about it? Well, if you enjoy sci fi movies and have seen some productions like The Minority Report, then you might like its features… Not only fun, creative, but also open source! In October 2013 Microsoft Open Technologies released the open source project Kinect Common Bridge for Kinect for Windows making it simple to integrate Kinect scenarios and experiences in creative development. MS Open Tech team has worked closely with open source communities such as openFrameworks (OF) and Cinder and also with the Kinect for Windows team to integrate KCB in popular creative development libraries and toolkits. image source: Microsoft Open Technologies blog  The Kinect for Windows SDK allows to use the Kinect for Windows sensor in a C++, C# or Visual Basic application with advanced control on the Kinect data. The Kinect Common Bridge sits on top of the Kinect for Window SDK providing useful and powerful abstractions allowing developers to focus on creativity and building great apps. The project is available on GitHub. New release, new features In the first update of Kinect Common Bridge, the Microsoft Open Technologies team made it easier to track faces and recognize speech in C++ applications with Kinect for Windows. Adalberto Foresti wrote: “In the spirit of “focusing on the cool stuff” that motivates creative developers, starting the sensor and displaying a simple video treatment with face tracking can now be achieved in less than 10 lines of code!“ that does sound great, doesn’t it? The latest update to V2 beta (which is to be found on GitHub) focuses primarily on enabling developers to quickly integrate the Kinect v2’s new sensor capabilities within a simplified set of C - based API’s. Kinect Common Bridge v2 complements the Kinect for Windows SDK v2, a set of resources designed to integrate Kinect scenarios into a variety of creative development libraries and toolkits. The Kinect for Windows v2 SDK new features: Window Store app development  Unity Support Improved anatomical accuracy Simultaneous, multi-app support: Multiple Kinect-enabled applications can run simultaneously.  You will find the details of these new features as well as resources on Github helping developers to integrate Kinect v2 sensor functionality into their code here and technical documentation and tools here. Who is this for? The Kinect Common Bridge has been designed for and with the help of creative developers, and interoperates with open source C++ toolkit for creative coding - openFrameworks (OF) and Cinder - a community-developed free & open source library for professional quality creative coding in C++. It is broadly used by developers creating advanced graphic applications with 3D animations.  Kinect for Windows SDK comes with support for C++ and C# development. You can also download the Kinect for Windows Developer Toolkit for guidance and samples to get started coding or visit the Kinect for Windows Dev Centre. What can you do with Kinect? That – we actually don’t know, because it depends very much on your imagination. But we can tell you what Tomasz Kowalczyk, a developer from Poland has done. Here are some of his projects: Kinsector – an app for the monitoring of rooms which sends an sms/mms notification to the user, Heremo – an application for rehabilitation which monitors performed exercisesand provides feedback about their quantity and quality, NatuLearn - an educational project based on the Kinect system allowing to combine learning with physical activity.  If you would like to learn more about how to use Kinect in your projects, check out some of his articles: Kinect SDK - Natural User Interface API Kinect SDK - determination of distances Kinect SDK – skeletal tracking Kinect SDK – Audio API Here is also our interview with Tomasz: #DevSpotlight: What Can a Developer do with Kinect?    It was with great help and input from open source communities that this project came to life. We look forward to working together, exchanging ideas and sharing our work. Stay with us for more! - The Microsoft Openness At CEE Team

Top 10 Openness Posts | July 2014