Unlocking the Value of Your Data


Over the past several years, the interest in “Big Data” has socialized the value of data to businesses.  There are a host of companies developing platforms and tools to help companies utilize their data.   While there may be “Big Data” skeptics out there, it’s impossible for IT executives to ignore the issue.  This is all to the good, as it means that businesses are starting to take a serious look at the opportunity to use data to drive decision making, improve operational efficiencies and increase revenue.   But companies that haven’t addressed these issues before find it difficult to know where to start.   Having worked with and led data analysis teams for many years, I want to share some experiences about what works (and what doesn’t).

The Problem with Business Intelligence

The traditional approach in IT organizations to business intelligence is to define a problem, fund a project to create a data warehouse, and then analyze the data.    With the pace of business today, this approach is no longer viable.   It simply takes too long.   The traditional approach often depends on business case approval and funding cycles, which automatically add time and overhead.    Worse, it also depends on precisely understanding the problem and the benefits of a solution (which are needed for the business case) – all before looking at any data.    The problem with this approach is that you often need to work with the data before you can fully understand what questions to ask and what the data is telling you.    A simple analysis can often tell you whether a direction is likely to be fruitful or not.    It’s better to figure this out quickly, instead of after months of working through the business case approval process.

Learning to Be Nimble

Recognizing this, companies have learned how to be more nimble.    They may create a flexible team of analysts with a broad mandate to go after problems in a specific domain, such as operational efficiency, marketing insight, or brand enhancement.    It’s likely that this team will set up a data integration platform, and will then go after the data they need.     While this is the right approach, ironically, it also ends up taking too long all too often.   Why?  One reason is that it takes time to get all of the data in one place.   In my experience, up to 90% of wall clock time can be spent gaining access to the necessary data, even when that data is owned by someone in your business, while it takes only 10% of the time to actually analyze it.     Why does it take so long to get access to the data?

The analytics team often finds a number of roadblocks in their way.  They need to find out what data exists, who “owns” it, and get their cooperation.   This can take months.   If the data doesn’t exist, isn’t being collected, or needs to be enhanced, it can take even longer.     Moreover, in a large company, there may be multiple groups trying to solve related problems, who aren’t even aware of each other’s efforts.  This results in duplication of effort, which is not only slow, it’s costly.

Creating a Big Data Program

There are many aspects of a comprehensive program that uses data to drive business value.   There are issues related to tools, governance, expertise, and culture – each of which is a topic in itself.   In this section, I’ll touch on three elements of a Big Data program that cut cross these areas.

Recognize that Data is a Corporate Asset

First, leadership needs to recognize that every bit of data generated by the business is valuable: it’s a corporate asset.   You need to manage it, just as you manage other assets.   Achieving this often requires a culture change.   In the past, the goal was to get a product or service out quickly.   Collecting data about the service to produce meaningful metrics was of secondary importance.   Perhaps, if budget was tight, you might have faced the question: do we really need to collect that data?    Instead, the default should be “what can I measure that might be useful?” and “what can I measure that will allow me to make things better over time?”   Then, measure it and collect the data.   You can’t improve what you can’t measure.

Bring Data Together Proactively

Second, it’s important to recognize that analysts can’t analyze data they don’t have.   Instead of waiting for a business case, proactively bring all of the data about your service and your customers together in a data integration platform.   I’ve recently been hearing this platform called a “data lake” or “data ocean,” which gives the right impression.   This promiscuous attitude to data integration has become much more feasible with low-cost distributed storage, as compared with large engineered data warehouses of the past.   There are a broad range of tools emerging that can be used to analyze data that is brought into the data lake. Streaming analysis tools may be needed when it is important to generate results in real-time.   Data search/exploration tools allow interactive exploration of both unstructured and structured data.    It’s valuable to create a meta-data repository about the data in the lake, so analysts know what is there, and can find the data they need.

Establish an Open Data Culture

Finally, cultivate an “open data” philosophy, and facilitate sharing of data, code, results, and approaches among your teams.  Clearly, you need to worry about information security, so that sensitive customer and business information is not vulnerable to attacks.   But sharing within the business should be the default, not the exception.  For example, you could mandate that all internally developed analysis and visualization software be stored in an internal Github repository.   Data analysts should also be encouraged to publish derived data or results via APIs that are documented in the meta-data repository.    In this way, when one analyst figures out how to join two data sets that lack a common key, all of the other analysts in your business can benefit from that learning.

By institutionalizing a culture that values data, it’s much more likely that the data that is needed will be collected in the first place.  By creating a data lake, meta-data repository and data exploration tools, you can eliminate or dramatically reduce the time spent getting access to data.    And, by encouraging sharing of data, code and results, new projects can readily build on the success of older ones.

This is not the traditional business intelligence approach.   It’s not about knowing what you want to do, in detail, up front.   The data will teach you way you need to know, if you create the culture and framework that allow analysts to unlock its value.

Posted in Technology | Tagged , , , , | 2 Comments

Start-ups Bridging the Virtual and Physical Worlds: Three to Watch from the January New York Tech Meet-up


Over 500 techies braved the Arctic chill last night to attend the New York Tech Meet-up last night, at NYU’s Skirball Center. Either the weather or the first demo solidified the theme for this article, which is technology start-ups that bridge the physical and virtual worlds. Perhaps I’m just rebelling against the Snapchats and Instagrams of the world. We spend so much of time in our virtual cocoons that it’s nice to see bright techies improve (and disrupt) things in the real world.

The first demo was by MakeSpace, which promises that you’ll never visit a storage space again. This sounds like a great idea to me. CrunchBase describes MakeSpace as a “next generation storage service company.” They offer a service that is ideally suited for New York City and other urban centers. In a nutshell, MakeSpaces drop off storage bins at your apartment, and you pack them and inventory what’s in them. MakeSpace then picks them up and stores them in a secure warehouse for a flat monthly fee. Voila! Technology comes into play through online scheduling, and your ability to manage your stuff online.

Key.me allows you to take pictures of your keys (front and back) and upload them to the cloud. Then, when you’ve locked yourself out, you call Key.me and they will either airlift a key to you, or send codes to a local locksmith who can cut a key for you. Key.me’s secret sauce is computer vision software that translates the digital image to data that can be used to cut a physical key. Pretty cool stuff. It turns out that there are lots of high security keys that can’t (yet) be copied by their algorithms. But 3D printers and more sophisticated algorithms will expand Key.me’s capabilities over time. In the meantime, they can grow their business and keep countless New Yorkers from sleeping in the hall.

The potentially most disruptive demo was by Oscar. Oscar is “a new kind of health insurance company that is using technology to make insurance simple, intuitive and human.” Oscar offers insurance through New York State’s health insurance exchange, and is the first health insurance company to be chartered in the state in over 15 years. Oscar’s web site says that they have over 40,000 providers in the Greater New York area already. Oscar is pushing the envelope on some radical ideas, like transparent pricing – both for their insurance and for health care services (that’s pretty radical) and free phone consultations by a doctor when you don’t really need to make a trip to the doctor. The Oscar team also demonstrated an intuitive natural language interface to guide customers to the health care specialist that can help them best. Oscar isn’t going to fix health care in America single on their own, but it’s nice to see someone do something exciting in the stodgy old business of insurance.

There were a bunch of demos that live mostly in the virtual world. Passomatic allows you to “change passwords in less time than it takes to say ‘woohoo’.” Koding is delivering “the coding environment for the future”, and has some nifty features aimed at distributed teams of developers. Docurated has created a content search/discovery tool that allows users to forget about drives, folders, and clouds – it searches all of your repositories, presents all of the relevant content in a stack ranked order based on relevance, and allows you to manipulate your content at the page or object level. PhotoFeed creates a unified photofeed across all your photo sources. Blogcast, the Hack of the Month, is a widget that allows readers to talk to their favorite blogger (available on GitHub). LiveCube provide an audience engagement tool for events, powered by game mechanics. The demos concluded with OneToday from Google. It’s a mobile app that facilitates community based giving, every day.

So we come full circle: back to the boring old real world, where an intuitive mobile app makes it easier for each of us to make a difference. Tech is not just about friend requests or funding rounds or the battle for the home screen. It’s also about using technology in creative ways to make a difference in people’s lives. Which, by and large, are lived in the physical world.

Posted in Technology | Tagged , , | 2 Comments

When There is No Enemy Within, the Enemy Outside Cannot Hurt You


Fostering a Culture of Innovation

A colleague who holds a senior position in a technology company recently asked me about innovation.    Specifically, his question was “How do I foster innovation internally, rather than through acquisitions?”     Since this is an important question for many business leaders, I thought I’d share some of my experiences with a broader audience.    In my view, creating and maintaining a culture of innovation is essential for companies to stay competitive, to grow their businesses, and to keep employees engaged.       Being good at it requires commitment across the organization, a willingness to experiment (and fail), and the courage to take a hard look at places where barriers to innovation exist.

In this note, I define innovation broadly, as ideas leading to significant improvements in products, services, technologies, processes, or business models.    Companies and organizations typically succeed by doing one thing exceptionally well, whether it is delivering consistently high-quality food, while managing cost; or delivering high-quality code on time and on budget.    Unfortunately, the heads-down focus that is necessary to achieve this is not always conducive to the behaviors that are needed for innovation.

Even companies that grew out of an innovative idea and a “start-up” mentality can stagnate as they grow and mature.    It’s even more difficult to create and maintain a culture of innovation in large companies that have an internal culture that has traditionally valued other things.    A focus on meeting targets can lead to risk aversion.  In large companies, there are often strict divisions of responsibility and heavy-weight processes for coordination among groups that create an inherent barrier to innovation.    And, leadership is often impatient.   It’s important for them to understand that any significant change will take time.

Other barriers to innovation include factors like management attention and attitudes, the way resources and funding are allocated, incentive systems, employee engagement and expertise, etc.     In many organizations that are struggling with innovation, the staff at the working level are not the root cause of the problem.    They have ideas and want to contribute, but may not have been given the opportunity.   This is why management attention and resources also need to be addressed. 

Framing the Problem

The first step is to determine what kind of innovation is needed in your company.     For an innovation to be successful, it needs to be aligned with the broad business strategy, address your customer’s needs, convey a competitive advantage in the market, consider the capabilities needed to execute, and deliver value to the business, such as revenue growth or customer retention.     These are issues that are best addressed by leadership, given that they have the broadest perspective on the business, though I find it can be very useful to get input on overall direction from working level staff as well.

Once leadership has set a direction, the next step in creating a culture of innovation is communication.    Leadership needs to clearly and consistently communicate what it wants to achieve throughout the organization, and back up this message with action.    Leadership may have some very specific goals in mind, may simply want to stimulate more internal innovation, or a combination of both.   For this discussion, let’s break it down in two possible objectives: directed (or focused) innovation, and bottom-up innovation.

Directed innovation is focused on specific problem domains where leadership knows it needs to be innovating.    Bottom-up innovation is more open-ended, and relies on a culture that encourages people to recognize needs, and to propose / create solutions to their needs.    These are not mutually exclusive.   For example, one of your employees may tell you it’s important to be innovating on X.    This can lead to a directed innovation work stream on X.

Let’s start with directed innovation.   The traditional approach would be to simply appoint someone to be responsible for the area, then allocate staff and a budget.     But good ideas can come from anywhere; sometimes from places you’d least expect.   How do you get the entire organization to own a problem and contribute ideas and solutions?    In a small organization, you’d engage everyone directly.    You might pull together a brainstorming session, frame the question, board all of the ideas that come out of the discussion, and then vet them to identify the best ideas.     Once you have the best ideas, you’d figure out how to act on them, hopefully in a way that engages and/or recognizes the people that contributed to each idea in the first place.   In a large organization, you need to do the same thing, but in a scalable way.      Management might issue a company-wide “challenge” in a given area, say customer service, and listen to what employees have to say.       Scaling this can be done any number of ways, such as through the management hierarchy or perhaps by crowd-sourcing ideas through an internal social web site.    More on this later.

What about bottom-up innovation?     How do you get people to innovate broadly as part of their job?    How do you capture ideas and vet them?    And how do you take action on the best ideas, engaging and/or recognizing the people who came up with the suggestion to begin with?   Like directed innovation, in a small organization, you’d ask everyone for their suggestions.    Or you might establish a company-wide guideline, as Google used to, that everyone can spend up to X% of their time on a project of their choosing.    But allowing everyone to spend time on a project of their own choosing is pretty open ended.   On one hand, that’s the point:   everyone should believe that innovation is part of their job.   On the other hand, with bottom-up innovation, you have the problem, not just of scale, but also of allowing of creativity to occur freely, while maintaining a degree of control over how much time is invested in it, and which ideas are acted on.   


Both directed and bottom-up innovation have the same intrinsic steps: communication, idea generation, vetting, and action.    Issues of scale and level of investment come into play in both directed and bottom-up innovation.   As there isn’t a formula for addressing the steps, I’ll describe some things that I have seen work, and that are generally applicable to many organizations.

Since ideas can come from anywhere, the challenge with communication and idea generation is how to encourage creativity, including how to reward people for coming up with good ideas even in areas that are outside of their day-to-day responsibility, and how to capture those ideas.     The first step is communication: letting people know what you’re looking for.    If line management has the right mind-set, they are your best advocates.  If an employee’s idea is something that he or she can act on unilaterally, or with support from line management, they should.   The message is that innovation can be stimulated simply by giving people the message that they are empowered to take action, rather than waiting for direction.

Focusing a bit more on idea generation, in some cases, for example in an area that cuts across functional units or if line management is part of the problem, you may want to create new channels of communication that don’t depend on line management.    For example, the CTO or someone designated by the CTO could convene a set of skip-level sessions for brainstorming, with participation either at random, or by having people self-select to participate.      Another approach to generating ideas outside of functional areas is to convene a group of top individual contributors from different parts of the organization in an Advisory Council.     This group can be given a mandate to produce its own recommendations outside of the normal chain of command.    As mentioned above, another approach that can capture valuable ideas in very large company is to create a social networking web site for crowd-sourcing ideas.    This web site can be used to issue management “challenges” to focus attention on specific problem domains.

Once you have a set of ideas that go beyond an individual manager’s purview – about a specific problem (directed innovation) or more generally (bottoms up innovation) — they need to be vetted.    Someone needs to decide which ones to invest in further and which ones to drop.    In some cases, the ideas may need to be developed before they are ready to withstand the vetting process.    Depending on his or her expertise, the innovator may need help fleshing out some parts of the idea.    In a large company, it could make sense to have a small, dedicated team available to innovators to help them put together their “idea pitch.”     The final decision on which ideas to pursue should be made by senior leadership.   Getting buy-in from the CTO and CMO is a good indication that an idea may have legs.     If the process you use gives innovators the opportunity to interact directly with the CTO and CMO, independent of line management, this creates its own incentive for people to participate.    Most people don’t get an opportunity to pitch an idea to the CTO or CMO on a regular basis.   When given the opportunity, people rise to the challenge.

The point of the selection process is to select ideas for further investment.   Up until this point, the innovator(s) have been working on this in their spare time, so the resource investment has been relatively modest.    The next step in a tech company might be to go from Powerpoint to prototype.   Once you get down to building a prototype, the project may need additional resources, and a project team may need to be created.

The expectation should be that members of the project team can dedicate a significant portion of their time to the project.    If an innovator and their management agree that the innovator can reduce his or her hours on day-to-day responsibilities without giving up that job, it may create a rotation opportunity to train someone else in that position, while the innovator is spinning up the project.      Alternatively, if the project is a larger undertaking and leadership is clear that the project direction is sound, it can be operated as an internal start-up, where staff is transferred into a new organization.     However, I think it’s best to view innovation projects as experiments, with a small initial investment, before turning them into mainstream development projects.    This allows the project team to be nimble and to learn by doing, before the technical and business direction is set.    It also controls the investment ramp.

Project teams should set clear objectives and there should be a regular cadence for reviewing status with leadership.    The investment may be relatively small in the first round of a project.    One can view the first phase of funding as seed funding from a venture capital firm.    At the end of each round, the innovator(s) need to come back to leadership to request more funding, and permission to move forward with the project.    Each decision point is an opportunity to tweak the direction the project is taking, or to cancel the project.

It’s also important, regardless of the process you use, that innovators get good feedback about their idea, and thanked for their suggestions.   Not every idea will be adopted.   Idea selection must be done in a way that is fair and is perceived as such.   For ideas that don’t make the cut, the rationale for rejection should be clearly articulated, so that rejection doesn’t create disincentives to future participation in the process.

Additional Things to Consider

The discussion above does not address gaps in expertise that exist in the organization, which may make it difficult to get disruptive innovation.   You can’t create in an area you don’t understand.    If there are areas that are important to the evolution of your business, you may need to hire some people in that area.    In the CTO organization, you might create an advanced technology group responsible for these emerging areas.   A partnership with another company can also be a valuable strategy, though this note focuses specifically on internal innovation.

Depending on the size and culture of the organization, it may be appropriate to give someone responsibility for spearheading innovation.      You need to pick someone with the right mindset, technical and inter-personal skills.   In a culture that is resistant to change, the CEO or CTO needs to give this person a certain amount of moral authority.   However, there are risks to codifying innovation in a particular position.   Innovation is everyone’s responsibility: you don’t want to make someone into a gate-keeper or elevate their role at the expense of the broader organization.

Deciding what behaviors and outcomes are desired is important.    People will respond to management direction.   To the extent that management doesn’t know what it wants, or is prone to suggesting a lot of unfiltered ideas that people feel like they have to pursue, it can stimulate a great deal of uncertainty and wasted effort.     This does no one any good.

Despite the challenges, however, I believe that old dogs can learn new tricks.    Companies can reinvigorate themselves and improve their competitive position by stimulating internal creativity and innovation.   To paraphrase an old African proverb: “When there is no enemy within, the enemy outside cannot hurt you.”

This note brings together the accumulated wisdom of a lot of people.   I hope that others find it useful.

Posted in Technology | Tagged , , , , | Leave a comment

Where is Google’s Robotics “Moonshot” Taking Us?


With Google’s acquisition of the robotics firm, Boston Dynamics – it’s eighth purchase in this space – robotics jumped into mainstream consciousness. If you weren’t paying close attention, your reaction was probably something like: “what’s that about?” If you were paying attention and watched one or two videos about Boston Dynamics’ robots, you were either fascinated, terrified or both. There is something both uncanny and deeply disturbing about the machines Boston Dynamics has developed. The best known is Big Dog, a quadruped robot designed for DARPA to serve as a pack animal for soldiers in rough terrain. The way that Big Dog ceaselessly prances, and recovers its balance when shoved, is both animal-like and machine-like. This amalgamation of attributes takes something comforting and familiar, and combines it with something alien and potentially threatening. I’ve had a similar reaction to people wearing Google Glass. Glass turns an innocuous bystander into someone who is vaguely threatening. My reaction to these technologies reminds me of the story about the early days of motion pictures. As the story goes, in 1895, when an audience first saw a movie of a steam locomotive coming straight at them, they screamed and ran to the back of the house. These technologies are taking us into places that we don’t thoroughly understand. It will take a while for the collective unconscious to catch up.

This is one reason that Google’s investment in robotics is so interesting. Google is playing the long game. Clearly, they’ve seen the huge advances in robotics over the last decade. Much of this progress is driven by a focus on highly specialized applications. The DARPA Robotics Challenge and Japanese investment following the Fukushima disaster have focused on disaster recovery (the 2013 DARPA challenge is underway in Florida this weekend). In a similar vein, robotics applications ranging from industrial automation on automotive assembly lines; to remotely operated vehicles for space or underwater exploration, explosive ordinance disposal, and cleaning up oil spills; to commercialized agriculture, have become routine even if they are unfamiliar to most of us. But Google the company is not one bit interested in specialized applications such as emergency responders or the military. Google focuses on appealing to consumers. It’s whole existence is because hundreds of millions of ordinary people use its search engine every single day, allowing it to make money as an intermediary between advertisers and their markets. A well-managed company makes investments that are aligned with its core mission. If you believe that Google is a reasonably well-managed company (and I do), its leadership believes that robots will eventually touch large numbers of people. They believe that robots will both generate and use vast amounts of data. And, they are willing to push the envelope on robotics well before our culture is ready for it. Like the proverbial field of dreams, if Google builds it, they will come.

Google brings assets to the table that enhance humanoid robots in very interesting ways. Your robot needs maps and navigation to get around. Check. It needs to be able to understand and speak multiple languages. Check. It needs to know everything about you, so that it can anticipate and better service your every whim. Check. None of this is ready for prime time, but Google is one of the few companies with deep enough pockets to dream this big. Nobody knows which commercial application will hit the marketplace first or which will succeed — a humanoid version of Roomba that handles all of your mundane tasks, or a modern Furby that is your Best Friend Forever. But today’s advanced robots are the stuff of science fiction made real. We can be frightened by the world that technology is making possible. But the nature of that world is still determined by humans, not humanoids – by engineers and consumers, policymakers and investors. It is up to humans to imagine and create a world we want to live in. I hope we’re up to the task.

Links to more videos: Petman, Bloomberg, and BBC.

Posted in Technology | Tagged , , , , , | 7 Comments

Mobile Customer Care: How Can Mobile Improve Customer Experience?


I was scheduled to participate in a webinar on mobile customer care last week, which was rescheduled to February. I’ve been involved in a number of customer care projects, working with both IT and customer care organizations that deliver these services at massive scale. Here are some of the lessons I’ve learned, which I will talk about when this event happens.

Customer Experience: The New Old Religion

Customer expectations about how they interact with the companies they do business with are changing, and successful companies are ones that have a customer-centric mindset in their DNA. Even companies that have neglected customer experience are getting religion, as competition becomes more fierce. Since today’s customers are mobile, and use a variety of mobile devices, a natural question to ask is how should mobile technologies be used to improve the customer experience.

Traditional Customer Care Technologies

First, it’s useful to look at more traditional customer care technologies. Traditional IVR systems are necessary, but can be a source of customer frustration. Self-service times for automated interactions can be quite low, as customers struggle to map their problem to a list of menu choices and simply press zero for an agent. Long holding times are a clear dissatisfier. Companies that use IVR analytics tools can significantly improve the situation, though it’s important to recognize that using the wrong metrics can also hurt customer satisfaction. If agents are rated on Average Handling Time, they will do what they can to get rid of callers quickly. On the other hand, tracking First Call Resolution, where agents are rewarded for actually solving the customer’s problem, is generally a good thing.

Customer self-service via the web has become the dominant means of interacting with most companies. Yet, customer experience on the web can still be less than perfect. For infrequently used accounts, the first step in the interaction is often userid and password recovery. Moreover, even with the web, the devil is in the details: the UX may have deficiencies when trying to resolve specific problems, and there are problems that just can’t be easily resolved on the web. When you introduce mobile, it makes some things better, but others worse. The immediacy of mobile is great, and automatic login is routine. On the other hand, screen size limitations and touch interfaces on mobile devices generally make the mobile experience less full featured than the web. As a result, it’s important to think carefully about what problems to solve in a mobile care app.

The Right Approach to Mobile Customer Care

There are several important points to think about when developing a mobile care strategy. I’ll cover highlights of each point.

First, you should ask yourself what unique characteristics of mobile applications can benefit your customers. You should take advantage of the visual display of smartphones for easy navigation and presentation of information. An application can provide custom alerts when launched that provide personalized and relevant information to the customer. Customers are accustomed to text messaging on mobile devices, so chat is a natural means of interaction when an agent needs to be involved. Finally, you can provide information about how to solve some problems by integrating video tutorials into the application.

Second, analyze the data you have about customer touches. If you use it, data you already have will give you a good understanding of the kinds of transactions that are most common, and the things that can be handled easily and efficiently in your mobile application. Which transactions are most prevalent? Consider putting them on the home screen in your application. Which transactions can be easily automated? Come up with a scenario that allows the user to handle them easily from their mobile device. Which transactions are likely to require an agent to resolve. For these issues, make it easy for the customer to go directly into a chat session or to place a call to your call center.

Next, it’s important to recognize that mobile care applications are not a one-size-fits-all solution. Not all customers have a smartphone, know about, or will use a mobile care application. Customers will utilize multiple channels, based on the nature of the problem, time of day, convenience, etc. If you are responsible for customer experience, your job is to promote the preferred care channel, while offering and optimizing the customer experience across all channels.

Finally, when you do get into the design of the mobile application, it’s best to have someone on the team that knows how to think about UX design. They can work with your IT, customer care and analytic teams to ensure that the mobile care application provide an stunning experience, and that it fits into the rest of your portfolio of care solutions. If you approach the problem in a holistic way, customers will want to use your mobile care application, and it will become a factor that contributes to a positive customer experience and to that ultimate goal: customer loyalty.

Posted in Technology | Tagged , , , , , | Leave a comment

Three to Watch from the December New York Tech Meet-up


I attended my first New York Tech Meet-up last night, at NYU’s Skirball Center. It was great to see an auditorium full of NY techies – a tiny indication of how dynamic the tech community in NYC has become over the last few years. The NY Tech Meet-up is focused on demos, rather than talks: New York’s senior senator, Chuck Schumer, introduced himself as the first Demo. Schumer made some very reasonable comments about policy issues affecting tech in NY, but the most impressive thing was that he stayed to see all of the demos. Most bigwigs leave right after they give their spiel.

The Instrumental Section

All of the demos were interesting, although three demos really struck a chord with me (pun intended). The first was Instrument 1 from Artiphon. Though it needs a new name, it’s a beautiful, hand-crafted hardwood “multi-instrument,” with a patent-pending fretboard interface and velocity and pressure sensitive strum section. It generates MIDI I/O and is designed to be played in five different positions. I am not a gadget freak, but I definitely want one. The compute core of the instrument is an iPhone or iPod Touch, which opens it up to the whole Apple developer ecosystem. Hardware start-ups are always interesting, partly because there are so few of them. This is the first instrument in a planned series, and was far and away the coolest thing at NYTM.

The Multi-media Section

I was also impressed by what Erick Schonfeld and Edo Segal of TouchCast are doing. These guys are promoting what they call the ‘video web,’ which allows a rich, interactive web experience inside a high-definition streaming video. Touchcast released its first product in June: a video editor (or TouchCast editor) that allows users to create interactive content with fully-functional web pages, HD streaming video, and other apps embedded in an high-definition TouchCast video. In addition, user-generated TouchCasts can be uploaded to the TouchCast web site. While it was difficult for the presenters to convey the full power of the idea in a few minutes, the ability to embed and interact with applications inside a streaming video was pretty neat. Of course, multicore processors and media processing instruction sets have only recently enabled compositing of multiple simultaneous video streams with other content. This is an idea whose time has come.

The Social Section

The third demo that I’ll mention is a new social networking app called Nextdoor that addresses what seems to me to be a real social need, at least in America, which is the loss of community. Nextdoor is a private social networking application for neighborhoods. Technically, the idea is simple, but the developers have worked hard on some key problems, such as address verification, to ensure that the virtual communities that are created correspond to the actual residents of physical neighborhoods. They’ve also thought about realistic privacy concerns, for example, by allowing people to list their street but not full address. I was invited to join a Nextdoor community in my neighborhood several months ago, and I can say that it is useful. I think Nextdoor has identified a valuable and unexplored corner of the social networking space.

Wrapping Up

All of the other applications that were demonstrated were interesting: Priori Legal (on-line search for legal talent), Canopy (easy access to medical interpreters), Hitlist (airline bargain notification), General Assembly’s Dash (web developer training), Shutterstock’s Skillfeed (on-line training), and WiredNYC (LEED score for broadband) . One that deserves an honorable mention is Eric Schles’ work with an unnamed government agency to use web mining to identify victims of human trafficking. This was the only demo that did anything with data – Eric has used both text and image analysis in his work. All in all, NYTM was a great event.

Posted in Technology | Tagged , , , , , | Leave a comment

Streaming in the Cloud: Another Arrow in Amazon’s Quiver


I just finished reading the developer’s guide for Amazon’s recently announced Kinesis service.   Kinesis allows customers to build applications that process data streams in real-time.   It has built in fault tolerance, and can scale up and down with data volume.

To a developer, Kinesis’s architecture is pretty straightforward: data producers generate continuous, ordered streams of data records.   Developers write Kinesis applications, which access the data using the Kinesis client library, and can process data streams in whatever way they want.    Streams can be sharded so there are no hard capacity limits for a stream (shards have a 1 MB/second ingress limit, a 2 MB/second egress limit, and up to 1000 PUT transaction per second.)  Since a stream can have multiple readers and writers, the service allows for several different types of processing to take place on a stream.     For example, one application might generate a real-time alert, while another application might pre-process the data before loading it into a database accessed by a reporting app.    A third Kinesis application might load raw or aggregated data into a long term data warehouse like Amazon Redshift.    Pricing is based on an hourly shard rate and volume of PUT transactions.

Kinesis is part of the wave of new tools designed to handle high-volume data streams, such as Kafka and Storm.    In fact, Kinesis combines some of the features of both, adding Kafka’s message persistence to the scalable, high-performance stream processing that Storm provides.    The “magic” in Kinesis – the logic that provides fault tolerance, replication, etc. – is all under the hood.     This is exactly the reason that Kinesis is likely to find a significant user base: it’s a managed service.   Kinesis handles a lot of the details of provisioning and management for you – and it’s tightly integrated with the rest of the Amazon’s cloud services.    You can use Auto Scaling for Kinesis applications, DynamoDB to stored derived data, and Redshift for long-term storage.    Right now, Kinesis is only available as a Limited Preview in the U.S. East region, so you may not be able to get started just yet.   However, if your data is being generated in Amazon’s cloud and you need to do streaming analytics, it seems likely that Kinesis will be the natural choice.    Over time, as open source Kinesis applications start to appear, developers will find it even easier to build complex streaming applications in the Amazon environment.   Overall, Kinesis is a valuable addition to Amazon’s arsenal that can only add to their dominance in the IaaS space.

Posted in Technology | Tagged , , | Leave a comment