NetKernel :

XML Processing forthe 21st Century

Brian SlettenBosatsu Consulting, Inc.brian@bosatsu.net

Speaker Qualifications

Over 14 years of software development experience

Has own software consulting company for design, mentoring, training and development

Currently working in Semantic Web, AOP, Grid Computing, P2P and security consulting

Pinky Committer

Agenda

What the heck is NetKernel?

Why We Care

NetKernel Overview

NetKernel Languages

Advanced Concepts

Pinky

What the heck is NetKernel?

A micro-kernel based resource ecosystem built around the principles of REST, Unix Pipes and SOA

What the heck is NetKernel?

A subversive, generalized architecture that allows you to prop up what other nonsense you are forced to use

What the heck is NetKernel?

A scalable replacement for your client web tier

What the heck is NetKernel?

A highly productive replacement for business logic and middleware tiers

What the heck is NetKernel?

An efficient replacement for your data layer and persistence tiers (Look, Ma, No Objects!)

What the heck is NetKernel?

A powerful orchestration layer for your SOA that allows you to work in your languages of choice

Why We Care: The Simple Answer

Most enterprise systems use XML and are built in Java and this is still too much of a pain

Complicated

Different Languages for different technologies

Predicting scalability

Why We Care : The Complicated Answer

This is hard because of a general mismatch at the intersection of languages, data model, processing abstractions, architectural tiers, etc.

What Do We Want?

We want a high-performance, scalable infrastructure for unifying how we refer to data, documents, services, resources, etc.

XML-processing is just the first use for this kind of an architecture

Larger Trends

1970s 1980s 1990s 2000s

Code written

Dataformatted

NetKernel

A Ground-Up Resource-oriented computing environment

Modern, microkernel architecture

Takes the best of REST, Unix Pipes, SOA

Open Source w/ Dual License for non-open source uses

Resource-Oriented Computing

Everything is a Resource

Everything is URI-Addressable (logical reference)

“Computation” is the turning of logical resource references into physical representation

Lossless conversion between representations

NetKernel Properties

Fine-grained caching

Self-optimizing based on use

“Aspect-Oriented” architecture

Security, logging, etc. behind URI interception

Tiny Footprint

Can do “real” work in a VM w/ 10-12MB

Takes full advantage of multi-core, multi-CPU boxes

REST

Representational State Transfer

Architectural style described in Roy Fielding’s Ph.D. thesis

URLs locate resources in Internet-sized address spaces

Client passes all state in request

Conventional Notion of REST

XML Document

http://localhost:8080/workbench/nfjs/hello.bsh

Executable Code

http://localhost:8080/workbench/nfjs/hello.xml

URLs ! => Scalability

Addressability of the resources is important, but REST derives its properties from the architectural constraints

REST Properties

REST Structure

Unix Influences

Everything is a file (common interface/resource abstraction)

Small, focused, reusable tools

Tools are scripted

Processes have context from environment

SOA Influences

Loosely-coupled

Asynchronous requests

Reusable services composed through orchestration languages

Putting the Kernel in NetKernel

Internally, a microkernel manages threading resources

Maps URI requests to managed resources and schedules them

General Concept

Calling a BeanShell Script

XQuery

XSLT

Active URIs

Internally, NetKernel uses a special URI naming scheme

A proposed IETF standard by HP and 1060 Research to encode complex URIs into standard forms

General formactive:<processURI>*(+<inputName>@<inputURI>)

http://ietf.mirror.netmonic.com/draft-butterfield-active-uri-01.txt

Sample Active URIs

BeanShell

active:xquery+operator@hello.xq+arg@foo.xml

XQuery

active:beanshell+operator@hello.bsh+arg@foo.xml

active:xslt+operator@stylesheet.xsl+operand@doc.xml

XSLT

NetKernel Benefits

“Deep” REST environment

Appropriate abstractionsFocus on information, not code!

Layered approach to building applications

Logical connections/requests get resolved at runtime (late-binding)

Basic NetKernel Tour

NetKernel Concepts(aka Really Cool Terms)

Accessors

Transports

Representations

Aspects

Transreptors

Accessors

URI-addressable component that provides access to NetKernel resources and services

Transport

A component that funnels requests from external sources into NetKernel URI requests (and ultimately Accessors)

ExamplesHTTP

SMTP

JMS

Representations

The ‘R’ in REST

In NetKernel, an object representing the resource

Exposed via immutable, cacheable “views”

Aspects

Not AOP “Aspects”

Immutable views of the underlying representation in a given form

ExamplesJDOM, DOM, String, BinaryStream

Transreptors

Fancy term for “transforming representations”

Transreptors return a resource in a specified form if conversion is possible

Useful for converting between Aspects

NetKernelLanguages

DPML

A simple NetKernel scripting language for specifying URI requests

Supports conditionals, loops, subroutines, etc.

DPML Sample

<idoc><seq><instr><type>copy</type><operand>ffcpl:/usr/local/NetKernel/modules/workbench/nfjs/hello.xml</operand><target>this:response</target>

</instr></seq>

</idoc>

BasicExamples

XQuery

XML Query Language

W3C Candidate Recommendation

Functional language

FLWOR (For Let Where Order By Return)

XQuery Sample

<abbeysayings>{ for $author in doc("active:xquery+operator@file:/usr/local/ NetKernel/modules/workbench/nfjs/quotations/ authors.xq")//* where $author/name/last = 'Abbey' return $author/quotation}</abbeysayings>

BeanShell

Supports embedded BSH interpreter

Java source interpreter

Interacts with other scripts, Java, etc.

Makes for very fast scripting

BeanShell Sample

void main(){ // Extract all speeches with lines // containing the parameter argument req=context.createSubRequest(); req.setURI("active:xquery"); req.addArgument("operator","abbey.xq"); output=context.issueSubRequest(req);

//Finally return response response=context.createResponseFrom(output); context.setResponse(response);}

“Real World”Examples

XSLT Input Data

XSLT Example

if( context.exists("this:param:param") ) { param=context.sourceAspect("this:param:param", IAspectNVP.class); mbid = param.getValue("mbid");} req=context.createSubRequest();req.setURI("active:httpGet");

url="http://musicbrainz.org/ws/1/artist/" +mbid+ "?type=xml&inc=sa-Album";req.addArgument("url", url);output=context.issueSubRequest(req);

sa= new StringAspect("<albums>" + mbid + "</albums>");req=context.createSubRequest();req.setURI("active:xslt");req.addArgument("operand", output); req.addArgument("operator", "albums.xsl");req.addArgument("band", sa);output=context.issueSubRequest(req);

XSLT Stylesheet

<xsl:param name="band"/> <xsl:template match="mo:metadata/mo:artist/mo:release-list"> <html> <body> <h1>Results for <xsl:value-of select="../mo:name"/></h1> <table border="1"> <xsl:apply-templates/> </table> </body> </html></xsl:template> <xsl:template match="mo:release"> <tr><xsl:apply-templates/></tr></xsl:template> <xsl:template match="mo:title"> <xsl:variable name="id" select="../@id"/> <xsl:variable name="url" select="concat('http://musicbrainz.org/ws/1/release/',$id)"/> <td><a href="{$url}?type=xml&amp;inc=tracks"><xsl:value-of select="."/></a></td></xsl:template>

XSLTExamples

Simple Tree Manipulation

Kind of a DSL for XML

XML is tree-oriented, let’s use it that way

STM

Sets element values to the specified value

stm:set

<stm:set select=”/x/y” xpath=”/a/b[1]/c”><x><y>zebra</y>

</x></stm:set>

<a> <b> <c food="hay">horse</c> </b> <b> <c food="hay">donkey</c> </b></a>

<a> <b> <c food="hay">zebra</c> </b> <b> <c food="hay">donkey</c> </b></a>

Inserts specified value after the selected element

stm:insert-after

<stm:insert-after xpath=”/a/b[1]”><x><y>zebra</y>

</x></stm:insert-after>

<a> <b> <c food="hay">horse</c> </b> <b> <c food="hay">donkey</c> </b></a>

<a> <b> <c food="hay">horse</c> </b><x><y>zebra</y>

</x> <b>

<c food="hay">donkey</c> </b></a>

Applies the specified namespace to the selected elements

stm:apply-ns

<stm:apply-ns prefix=”ns” uri=”http://www.bosatsu.net/ns”xpath=”/a/b[2]”/>

<a> <b> <c food="hay">zebra</c> </b> <ns:b xmlns=”http://www.bosatsu.net/ns”> <ns:c food="hay">donkey</ns:c> </ns:b></a>

AdvancedConcepts

Asynchronous requests are easy in NetKernel

Asynchronous Requests

// Set up request 1req1=context.createSubRequest();req1.setURI("active:beanshell");req1.addArgument("operator", “script1.bsh” );req1.addArgument("param", param );handle1 = context.issueAsyncSubRequest(req1);

// Set up request 2req2=context.createSubRequest();req2.setURI("active:beanshell");req2.addArgument("operator", “script2.bsh” );req2.addArgument("param", param );handle2 = context.issueAsyncSubRequest(req2);

// Block until both calls finishretValue1 = handle1.join();retValue2 = handle2.join();

Throttling requests is also easy

Throttling Requests

// attempt to issue 100 concurrent requests to script1.bshfor (int i=0; i<100; i++) { req=context.createSubRequest("active:throttle"); req.addArgument("id","throttle:my_throttle"); req.addArgument("uri","active:beanshell"); req.addArgument(“operator”,”myscript.bsh”); context.issueAsyncSubRequest(req);}

<throttle>  <concurrency>4</concurrency>  <queue>50</queue></throttle>

Expects to find a configuration file in ffcpl:/etc/ConfigThrottle.xml

Golden Thread Pattern

Virtual Dependency Management

Powerful technique for invalidating arbitrary collections of resources

Easier to see than hear about!

Long Computation wrapped in an Accessor called from a DPML script

Scalability - Scenario 1

Shorter Computation called multiple times from DPML script

Scalability - Scenario 2

Random document chosen from DPML script

Scalability - Scenario 3

Real NetKernel Infrastructure is built around modules

Modules

Ways of segmenting Address Space and functionality

Modules export public URIs and class packages

Separately Versioned units of deployment

Modules (cont)

Modules (cont)

Can hold static resources, Java code, scripts, accessors, aspects, transreptors, transports, fragmentors and caches

Import module definitions to access another namespace

If you create a new module, you will need to at it to the deployedModules.xml

Sample Modules

Module Functionality

ext-xml-core-1.4.2.jar DOM, JDom, SAX, XDA, XMLBean

ext-dpml-1.2.2.jar DPML Language runtime

ext-kernel-1.0.6.jar System component accessors

doc-netkernel-4.0.0.jar NetKernel Core Documentation

mod-fulcrum-frontend HTTP Server FrontEnd Fulcrum

mod-fulcrum-backend HTTP Server Backend Fulcrum

tpt-http-2.1.8.jar HTTP Transport

tpt-jms-1.0.2.jar JMS Transport

app-address-book-1.1.4.jar Address Book application

Module Example

Pipes in NK (Yahoo)Module for implementing Yahoo Pipes-like functionality

Runs in your computational space

No ads

Full control

Pinky

http://sourceforge.net/projects/pinky

Union - combine one or more feeds

Truncate - keep things manageable

Unique - remove duplicate stories

Count - how many elements per feed

Filter - restrict feed elements to filter criteria

Pinky Accessors

Pinky Example

Sophisticated management consoleSystem Healthcheck

Manipulate Modules

Monitor Memory, Throttle Settings, Workload

Edit Kernel Parameters

NetKernel Administration

http://localhost:1060/ep+name@app_ext_introspect_controlpanel

Non-XML Examples

NetKernel Links

Open Source Site http://www.1060.org

Commercial Site http://www.1060research.com

Article http://tinyurl.com/ghxg3

Front-End Fulcrum http://localhost:8080

Back-End Fulcrum http://localhost:1060

Slides http://www.bosatsu.net/talks/NetKernel.pdf

Tutorial http://www.bosatsu.net/talks/examples/netkernel.txt

Talk Links

Slides http://www.bosatsu.net/talks/NetKernel.pdf

Examples http://www.bosatsu.net/talks/examples/NetKernel-Examples.zip

Please WriteReviews

Feedback/Questions:brian@bosatsu.net