Jan-Henrik,
Found some time today to do more testing on a problem I was still having with the mysql driver (the postgres driver is kicking ass).
I suspect there is still a problem in ensureCapacity (or it's consumers). I'm attaching a small program that will reliably reproduce this.
Looks like strings from the /first/ row (and only the first row) retrieved from a resultset are still being truncated at 256 char length.
tested against yesterdays svn-rev.
thanks,
Thanks Paul, I'll look into this over the weekend.
On 25. april. 2008, at 11.56, Paul J Stevens wrote:
Jan-Henrik,
Found some time today to do more testing on a problem I was still having with the mysql driver (the postgres driver is kicking ass).
I suspect there is still a problem in ensureCapacity (or it's consumers). I'm attaching a small program that will reliably reproduce this.
Looks like strings from the /first/ row (and only the first row) retrieved from a resultset are still being truncated at 256 char length.
tested against yesterdays svn-rev.
thanks,
-- ________________________________________________________________ Paul Stevens paul at nfg.nl NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands________________________________http://www.nfg.nl #include <stdio.h> #include <string.h> #include <assert.h>
#include <URL.h> #include <ResultSet.h> #include <PreparedStatement.h> #include <Connection.h> #include <ConnectionPool.h> #include <SQLException.h>
/*
CREATE TABLE `test` ( `id` int(11) NOT NULL auto_increment, `data` longblob NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=latin1
*/
const char *blob1 = "From nobody@pacific.net.sg Tue Dec 04 19:52:17 2007\n" "X-Envelope-From: nobody@pacific.net.sg\n" "Received: from [127.0.0.1] (port=49353 helo=test11)\n" " by centos.nowhere.com with smtp (Exim 4.63)\n" " (envelope-from nobody@pacific.net.sg)\n" " id 1IzWJv-0000Ep-5f\n" " for wallace@nowhere.com; Tue, 04 Dec 2007 19:52:17 +0800\n" "From: "Wallace" nobody@pacific.net.sg\n" "To: wallace wallace@nowhere.com\n" "Subject: Test 11\n" "Message-Id: E1IzWJv-0000Ep-5f@centos.nowhere.com\n" "Date: Tue, 04 Dec 2007 19:52:16 +0800\n" "\n" "\n" "This line works, however,\n" "From what I know, this line gets truncated\n" "This line gets truncated\n" "This other line get truncated too\n";
const char *blob2 = "("Tue, 06 Aug 2002 19:54:41 +0200" "[dovecot] mbox support" (("Marcus Rueckert" NIL "rueckert" "informatik.uni-rostock.de")) ((NIL NIL "dovecot-bounce" "procontrol.fi")) (("Marcus Rueckert" NIL "rueckert" "informatik.uni-rostock.de")) (("dovecot mailing list" NIL "dovecot" "procontrol.fi")) NIL NIL NIL "<0000420020806175441.GA7148@linux.taugt.net
")";
//#define DMTEST #define BUFSIZE 8192 int main(void) { char *in = blob1; int ZBDEBUG=1, i = 0; ResultSet_T res; PreparedStatement_T s; URL_T url = URL_new("mysql://test:test@localhost:3306/test"); assert(url); ConnectionPool_T pool = ConnectionPool_new(url); assert(pool); ConnectionPool_start(pool); Connection_T con = ConnectionPool_getConnection(pool); assert(con);
TRY Connection_execute(con, "DELETE FROM test");
s = Connection_prepareStatement(con, "INSERT INTO test (data)
values ( ? )"); PreparedStatement_setString(s,1,in); for (i=0; i<20; i++) PreparedStatement_execute(s);
res = Connection_executeQuery(con, "SELECT id,data FROM test LIMIT
10"); while(ResultSet_next(res)) { const char *out = ResultSet_getString(res,2); if (strcmp(in, out) != 0) { printf("Error mismatch\n[%s]\n[%s]\n", in, out); } else { printf("Row matches\n"); } } CATCH(SQLException) printf("SQLException: %s\n", Connection_getLastError(con)); FINALLY Connection_close(con); END_TRY;
return 0; }
-- To unsubscribe: http://www.tildeslash.com/mailman/listinfo/libzdb-general
On 25. april. 2008, at 11.56, Paul J Stevens wrote:
I suspect there is still a problem in ensureCapacity (or it's consumers). I'm attaching a small program that will reliably reproduce this.
Looks like strings from the /first/ row (and only the first row) retrieved from a resultset are still being truncated at 256 char length.
Darn and double darn! this seems to be caused by an ugly bug in MySQL, see http://bugs.mysql.com/bug.php?id=33086 This is a serious problem as it affect the design of the library. The libzdb library and the MySQL implementation is designed to scale well and not slurp down the full result set as this is just not feasible for large sets.
The method ensureCapacity() in MysqlResultSet.c is used to expand a column buffer as needed, and the function mysql_stmt_fetch_column() is used to re-fetch column data for the just expanded buffer. It turns out that MySQL does not update the buffer on re-fetch which is a serious bug in MySQL. Since the buffer has been expanded the next call to ResultSet_getString() will work in this case, but the first fetch will be truncated.
I'm really not sure how to best handle this in libzdb and I really don't want to store the full result set on the client. I will have to think a bit more on this and see if a work around can be found. Any suggestions are welcome. The bug was reported for 5.0.x and I got it in 5.1.22 on OS X. If you have a 4.x version of MySQL around, maybe you could test if its there as well? Not that it helps much as libzdb should work for all 4 - 6 versions.
Ps. As I said someplace in a code comment, MySQL seems to be developed and coded by the Swedish chef from the Muppet show.
On 25. april. 2008, at 11.56, Paul J Stevens wrote:
int ZBDEBUG=1, i = 0;
Just a quick FYI. I noticed that you use this to enable debugging in libzdb in all your example programs. ZBDEBUG is a global value in libzdb and the correct way to enable debugging is simply to do 'ZBDEBUG = 1;' Not to redefine the variable in your own program.