[These are notes from a session at 4D Summit 2008]

Presenter: Sergiy Temnikov, 4D SAS

It's important to realize that the SQL engine is not "on top of" DB4D, it's fused into it. There's definitely no penalty for using SQL.

v11.3 is a lot more than a dot release - there are a bunch of huge improvements to the engine.

• String searches 1.6x faster, numeric is 2 times faster,
• Getting the data out of the db into your arrays is 1.3 times faster,
• Numeric sorting is 1.2 times faster,
• Expression searches are 1.1 times faster,
• Numeric grouping (GROUP BY) is 1.2-1.3 times faster,
• String grouping is exponentially faster (!) depending on the number of records
• Min, Max, Avg, Sum, Count is 1.2 to 2 times faster

Connectivity options...

Previously...

• ODBC LOGIN (DSN) - traditional, requires configuration
• ODBC LOGIN (SQL_INTERNAL) is using things that look like ODBC, but it's not actually using ODBC.
• USE EXTERNAL DATABASE

v11.3 has simplied it - now it's just SQL LOGIN / SQL LOGOUT. They're renaming the commands so there's just the one entry point. Takes the same parameters as ODBC LOGIN. USE EXTERNAL DATABASE is still there, but deprecated - there's now a 4th parameter to SQL LOGIN that should be used instead. (BTW, DSNs can be preceded with ODBC:.)

v11.3 also allows direct/native SQL connections from 4D (anything) to 4D Server. You pass 4D:My_4D_Server (which is why ODBC: is needed for ODBC). Or you can pass something like IP:122.168.12.15 and just pass the IP address. Going native means no ODBC overhead. It's the fastest way to go 4D to 4D - 2 times faster than ODBC to login, and 3-10 times faster for data fetching, bulk inserts (with arrays) are 2-5 times faster. On top of that the error handling is far better (the entire error stack is streamed to the client).

And on top of that the 4D to 4D connection is secure, even without SSL. Every message is signed so they can't inject anything by pretending to be you, and usernames and passwords are not sent in clear text. HOWEVER, they can see the data. If you want to avoid that, use SSL (preference check box).

GET LAST SQL ERROR has been changed to GET LAST ERROR STACK and gives very detailed errors when doing 4D to 4D SQL connections.

SQL access rights have been beefed up in v11.3 as well. In v11.2 you have three groups of users with global rights throughout the database - read only group, read write group, and full access group. v11.3 allows you to have different access rights for different tables. It uses SQL schemas to do this. Every table must belong to one and only one schema, so schemas are groupings of one or more tables. They have implemented DEFAULT_SCHEMA and assigned all tables to that schema that makes the transition from 11.2 to 11.3 completely easy. If you want the schema can have just one table so you can have table-based access rights if you want. There is a new table in the system catalog that describes all the schemas.

There is a component available that will show all the schema in your database visually, but you have to set up schemas using SQL commands. The component will write the SQL statements for you. Once a schema is set up you can see/choose the schema in the table properties dialog. When you grant rights you grant them to a group, not an individual user and its done through SQL code.

People without rights can see that tables exist, but can't get the data in them.

On SQL Authenticate has changed. Previously you just set $0 to true. Now you need to call CHANGE CURRENT USER and the call has to be successful.

There is also a new multi-row insert capability.

There is also a new INFILE command to pull data from a file which is faster than loading the file into a variable. You can import text, pictures and blobs using INFILE.

You can also put BLOB data into picture arrays. This way you don't have to handle blob data differently than other data when pulling data into arrays. Just use PICTURE TO BLOB ($pict;$blob;".4DBlob") to get the blob out of the picture. However, if you have very large blobs it's still better to get them one by one since you may not need the blob data.

[These are notes from a session 4D Summit 2008.]

Presenters: Laurent Ribardiere, Laurent Esnault, Olivier Deschanels, 4D SAS

Laurent Ribardiere

What's changed since last year's Summit...

LR wanted to start by clarifying something about blob and text storage outside the record. There are times when it's a negative since when you have to search that field it's something like twice as slow. But then again if you don't need to do that the record is much smaller and more efficient. You can set the threshold as to when it's stored in the field and when it's stored outside.

Auto increment is much better than using a trigger for set key values.

TRUNCATE TABLE is extremely efficient way to dump data - something like 1000 times faster than DELETE SELECTION. They've optimized DELETE SELECTION so it uses TRUNCATE TABLE in certain cases. But do watch to see if it's failed since if it has problems with one record it does nothing. Also, it won't work in transactions.

The joins and how they work in QUERY BY FORMULA were improved in v11.2. You can now do things like:

QUERY BY FORMULA ([Employee];[Employee]CompanyID=[Company]ID & [Company]Name="4D")

A join is performed, there is no need for a structural link.  If you leave out the [Employee]CompanyID=[Company]ID a structural link is required. This may create problems because you may have intended to use the value of [Company]ID, not specify a join relationship.

The cache manager in v11 is much more efficient. No compaction is needed so it's far more scalable and can use non-contiguous memory. 4D v11 is a 32-bit application, so the theoretical limit is 4GB. The practical limits (from testing) are 1.5 GB on 32-bit Windows, 3.7 GB on 64-bit Windows (though not recommended to go that high), and 2.7 GB on OS X 10.5 (Leopard).

Even though 4D is a 32-bit app, there are benefits to running under a 64-bit system - namely the cache size can be larger. Basically under a 32-bit system you get a total 4 GB that can be allocated to 4D, whereas under 64-bit you get up to 16 GB with some things being stored outside the 4 GB that's used directly by 4D. A 64-bit version of 4D is coming... Then you'll get a total of 32 GB of RAM that can be used for 4D.

WAN performance for client/server is dramatically increased. Fewer requests and larger block sizes are the way it was achieved. QUERY went from 4 commands to 1, READ ONLY went from 1 command to 0. He did a demo on v2004 that took 34 seconds, on v11 it took 6 seconds.

Displaying a selection of records has also been improved over a WAN. v2004 had improvements over v2003, but v11 takes them further. The entire page is returned in one request and only the data displayed are returned. Listboxes are even more efficient than output forms because there are fewer tricks to get them to work.

Sets and named selections stick to where they are used (client or server). $ or <> are now used for namespace only. You could have a $ set on the server and a <> on the client (only). 4D knows when it needs to transmit back and forth and only does it when necessary. It's all transparent to the user and the developer.

Laurent Esnault

Client/server is now mostly based on polling, but on push. This is true in the development environment as well. Both the client and the server push as needed. For example the server could push an EXECUTE ON CLIENT out to the client.

You can log all the requests received by the server. This includes the Application server, DB4D Server and SQL Server requests. Activate it by doing SET DATABASE PARAMETER (4D Server Log Recording (28)) or "Start Request Log" in the 4D Server console. Olivier showed the logging action with a tail -f command in OS X...

QUERY BY FORMULA and ORDER BY FORMULA were previously discouraged in v2004. In v11 they're fine to use. It's now just one request to the server - it's no longer sequential over the network. Variables used directly in the formula are evaluated on the client. Variables used inside a method are evaluated on the server. Be careful that this may break compatibility - but there's a checkbox to make it act like v2004. If you need to turn this off and on dynamically use SET DATABASE PARAMETER. Compatibility mode is still significantly slower, but it's much faster than before because they page 100 records at a time instead of sending one by one.

Triggers are executed on the server in the twin process. Twin processes share transaction state and record locks, however there is no more client/server synchronization for current selection and other table records. The twin processes now have their own set of process variables in compiled mode. The execution of triggers no longer blocks other processes - many triggers can execute at the same time. You can now do pretty much everything in a trigger that you can do in a regular process except change the current record.

You can use Execute on client to push information to a client. The function is more efficient than it was previously.

You now need to think about when you want to use Execute on server (stored procedure) and when you just want to use the checkbox Execute on Server that's now a property of all methods. Execute on server (stored procedure) starts an independent process on server. You may find the Execute on Server (property) to be more useful and just as fast because it shares your process variables as well as $0.

Think about using the Execute on Server property to avoid triggers.

But make sure you don't just always execute on server since it may overload the server and not make use of the processing power of the clients.

Laurent Ribardiere

In terms of the architecture. The application server (4D language code on server, port 19813) and the HTTP/SOAP/Web server are still cooperative, single threaded. However, the SQL Server (port 19812) and the DB4D Server (port 19814) are both using preemptive multi-tasking.

The point is if you need performance stick with preemptive threads, not cooperative threads. That means you should keep the amount of 4D code running on server to a minimum (including triggers). If you call SQL from a 4D method it's cooperative. If you call it as an external data source it's preemptive.

They're working on making everything preemptive.

Do realize that you only have one data file and possibly many preemptive threads. 4D's cache is important, even better is the processor cache, but when those aren't enough contention can be an issue. So the more RAM you have, the bigger 4D's cache, and the faster your drives, the better...

DB4D data access is cooperatively threaded in 4D mono but preemptive with 4D Server. This can make somethings much faster. He did a demo showing how big a difference this can make. He also showed 4D using multiple cores on an 8 core machine.

[These are notes from 4D Summit 2008]

Presenter: Tim Kaufman, 4D Inc.

One of the problems with Flash was that the development environment felt alien to programmers. Flex fixes that - it provides an IDE which feels natural to developers, but compiles down to .swf files. Flash Player 9 is required to run Flex because it uses ActionScript 3.

The upside to Flash being proprietary is that runs in the Flash player which works the same across all platforms. Technically it's now an open standard...

Flex Builder is a full-featured IDE based on/compatible with Eclipse. It has a hybrid language of MXML and ActionScript. MXML looks sorta like HTML or XML, ActionScript 3 is a cousin of Javascript (both are versions of ECMAScript) though it has strongly typed variables.

It has VBoxes which are similar in concept to divs in HTML and one of the basic building blocks of Flex.

Flex gives you a lot of powerful features like sliders, drag and drop, etc...

Flex is completely object oriented (not just object based like Javascript). ActionScript has classes with properties and methods, for example. It's also asynchronous. Even the events are objects.

Data binding reduces the coding. So a destination object can listen for changes in a source object.

In terms of connecting to 4D. You can connect via web services. Flex is used more for presentation and less for business logic. Now with Flex for 4D you have a direct SQL connection and it brings it back into a dataset you can manipulate. This moves the business logic to Flex and out of 4D.

Using 4D for Flex to connect to 4D Server requires a fair amount (but not a huge amount) of object oriented programming - it's not a design/GUI type of thing.

4D for Flex is just a data provider, so you can use a number of different ways to handle the data once it's been returned.

Flex has a built in data grid which you can bind the data to. 4D for Flex has a data grid which extends the Flex data grid and gives you more functionality.

Once you get the hang of it he finds it incredibly productive and powerful - especially for prototyping. He prefers it over Ajax. An audience member said Flex thinks more like 4D than Ajax. ActionScript is the ideal intermediary between Javascript and Java.

It can't be run on mobile devices.

In the future they're planning on using HTTPS to do authentication, then they get a ticket, then pass the ticket in the SQL call.

[These are notes from 4D Summit 2008 - pictures to follow later.]

Presenter: Tom Fitch, 4D Inc.

Demoed a database with interactive SVG areas. Click on an area on the graphic (a seating chart for an arena) and it maps it to the appropriate data elements and lets you take action on that element (buying a ticket for a particular seat). Likewise the image is data driven, so the seating chart updates automatically as things change.

SVG is completely Scalable, 'cause it's Vector Graphics... That means no jaggies or blurred images. Also generally smaller.

The cool part is that you can do mouse events on a scalable area, which is really difficult without SVG.

Shapes and gradients are perfect for SVG. But complex images can be larger. However, SVG are uncompressed images, so if you're using gzip on the web it can dramatically reduce the file size making it more competitive with small raster images.

Labels and descriptions in the image mean they're searchable. Active events and triggers are possible, as is animation (in part or the entire image). To animate ECMAScript and SML(?) are used, but animation is not supported by all browsers (not supported by Safari or Netscape).

IE only supports SVG if you have the Adobe plugin for SVG.

SVG is a text-based open standard for images developed by W3C with the help of Adobe. It's also an XML specification.

Basic SVG shapes (more are possible):

• <rect>
• <circle>
• <ellipse>
• <line>
• <polyline>
• <polygon>
• <path>

<path> is extremely useful since it can draw non-standard shapes. It draws by specifying the points... d="M 100 100 L 300 100 L 200 300 z" - the first one is the starting point, then L draws a line to each new area and z takes you back to the starting point.

SVG can go into any picture area just like other image type. If you put your charts in a picture area 4D will write it as SVG. You can also build by writing xml with the DOM commands. DOM EXPORT TO PICTURE writes the xml to an SVG picture.

One thing to realize is that the stroke width goes on the outside of the area, not the inside.

To get On Clicked events you have to set an ID attribute. Then you need to GET MOUSE coordinates and then pass those to SVG Find element ID by coordinates, but make sure you subtract the position of the picture area from the values you get from GET MOUSE since you need the position inside the picture area, not the position inside the window. You can get the coordinates of the picture area with GET OBJECT RECT. You can also group objects in an SVG area and when you click on them they return the ID for the group, not the particular object.

One word of warning - you'll need to use keyboard shortcuts when tracing an On Clicked event until you hit GET MOUSE since the mouse position at the time of GET MOUSE is critical.

You can have raster objects in the SVG area.

Converting to PDF right now only on Mac because Windows doesn't have native support for PDFs.

[Notes from 4D Summit 2008.]

Christophe Keromen, Senior Product Manager, 4D SAS

To meet changing demands 4D will be focusing more on RIAs.

They'll also focus on collaborative development.

The infrastructure is changing as well... Wi-Fi, mobile computing, etc... The Ajax framework is important there as well.

4D will target 64-bit

Require fewer resources, and focus on mobile friendly technologies

They understand the need to soften the learning curve for current developers. Green Tea is a good example.

Revision and maintenance is the bulk of the apps lifecycle. As a result they can bring down the cost of ownership by focusing on lifecycle management.

They're coming out with a "Solution Manager" that will be a replacement for v11 Explorer. This comes from the fact that you can have apps with no data. The beginning of project management.

Version control is coming! Items will be externalized out of the 4DB file. Methods will be text files on disk. Forms will be external binary files. Other items will be externalized as well. The goal being to externalize everything.

There will be the concept of projects which will be much broader than the concept now. A solution will be one or more projects.

There will be support for multiple languages in 4D. Just as you can switch to SQL now, there will be support for even more languages.

Components can be edited while running the host database.

There will be aliases in groups, meaning an object can be a member of multiple groups.

There will be a filterable list of objects. The filtering will be pretty advanced.

You'll be able to edit a method in the preview.

More than than 91% of developers use source control - it's expected these days.

Version control will include Checkout, Update/Sync, and Checkin/Commit.

Since 4D is committed to open standards, version control for 4D will use industry standards. 4D's solution will be compatible with MSCCI (Microsoft's solution).  This means compatibility with commercial solutions and open source solutions (on Windows).

For OS X (and Windows) there will be 4D Server integrated version control. It's automatic with no configuration needed. It will require 4D Server. It will include a local repository so changes can be committed after they've been tested so other developers aren't affected by new bugs.

A new code editor is coming as well... Bigger font, smaller font... Folding has been improved. Find functionality is improved - highlights everything matching the text you enter, or just full words, etc. There's a bookmarking feature that then lets you navigate through the items that are found. When you have compiler errors there will be a special bullet in the sidebar that when you hover over it will tell you the error. Further, bullets will move when you add/remove lines above the bullet.

You can also use the code editor to edit Javascript, XML, HTML, PHP and CSS. A huge improvement is the ability to edit 4D web templates - so an XML file with the 4D web tags in them, etc. You'll have colored syntax, auto completion and folding at a minimum for each.

The next version will have SQL Views - basically virtual tables.

SQL server will continue to get better with each incremental release.

JSON (Javascript Object Notation is a way of describing data) will be supported in the next version. People who use it talk about AJAJ rather than AJAX... It's an easier way of dealing with data with Javascript.

They're going to implement a way Object Relational Mapping (ORM) so you can define a concept (an "entity model") from a business logic point of view so users don't have to be aware of how the data are stored relationally.  For example: Employee.Department.Name or Employee. Emp_MeetingList[1].Meeting.When Employee.EmployeeDetails.birthdate - much simpler way of thinking of data relationships.

4D will be able to generate a SQL view from an entity model and vice versa.

Everywhere you can use a table you can use an entity model instead.

Their RIA Server will support all the basics - HTML, Javascript, CSS, SWF (for Flex) but can serve data via XML or JSON.

The HTTP Server will have session management, DAF integration, PHP code execution. The HTTP server will be rewritten and based on Apache (but he didn't say that ).

They will improve existing method-oriented services as they did v11.2 and v11.3 with keep-alive and http compression. However, they're bring out new data-oriented services (XML, JSON, RSS) and based on Entity models. It will be based on the REST architecture. So the format will be specified in the URL (something like &format=rss or &format=json or &format=xml).

Not everything can be done in a browser, so they'll continue to innovate with client/server and standalone apps.

"Don't ask when"...