Better communication through introversion and solitude

I’m back. Sort of.

I’ve been busy rediscovering introversion, silence and solitude. Some activities require them, like learning, coding, learning to code.

I’m not going to discuss the paradox of coding as both a solitary and team/community activity, or get into the topics of communication, documentation, teamwork, that is for later.

I was inspired, among other influences, by a TED Talk by Susan Cain entitled “The power of introverts”  which is short and very interesting.

.This world we live in is very concentrated on appearances and clustering and peer pressure an that can dumb you down and disperse your energies. But that is also part of being human, since us humans have traveled a long way since we have discovered the advantage of hunting in “teams” for our survival. We have been tribes, guilds, nations, and lately even target groups divided by taste and purchase habits.

We, as humans, need to “hunt in packs” but we also need to move away from the tribe sometimes and invent something useful for everyone. Or even simply have fun playing with our Lego or Code.

Plus I need to rediscover what I need. And right now part of that is to read, learn and code.

I’m coming up with a way to keep using this blog or some other blog as a tool compatible with introversion and all of that.

For instance, the more code I write and more things I learn by myself, the more interesting my blog posts and tweets and other communications will be. I will have deeper and more concrete words to share.

So, see you later.

Behind the questionmark in a diamond – what I learned today about character encodings

I used to dread the days when I ran into a question mark, in a black diamond shape or not, in a middle of some text on a website or elsewhere, but now I find it fun. I feel like I’m finally getting the grip on the character encodings question, and I also like the idea of looking behind the scenes into what is really going on inside a file, or a program, or a process, or a database, a website, or anything else. Well, one tiny little baby step at a time, of course.

Today I found a question mark in a pdf, and wanted to know why. I went into the database to look at the text that was being printed in pdf, and just intuitively copied the text to then paste it into the shell and do a hexdump on it. I didn’t know if the copy-paste would modify it in some way, but I thought I’d give it a shot and see what happens.

I pretty much did this:

$ echo “Example of a possible phrase: single right quote’s problem” | hexdump -C

00000000  45 78 61 6d 70 6c 65 20  6f 66 20 61 20 70 6f 73  |Example of a pos|
00000010  73 69 62 6c 65 20 70 68  72 61 73 65 3a 20 73 69  |sible phrase: si|
00000020  6e 67 6c 65 20 72 69 67  68 74 20 71 75 6f 74 65  |ngle right quote|
00000030  e2 80 99 73 20 70 72 6f  62 6c 65 6d 0a           |…s problem.|

That is UTF-8 Unicode for you. It is backwards compatible with ASCII, so where it can it uses just one byte for character. And this one strange character took 3 bytes. To make the long story short, when we were passing (the choice of coders other then us) this nice UTF-8 to the php function utf8_decode() it was turning it into single-byte ISO-8859-1. And losing the right single quote. Because as I was told today, if you translate something complex into something simpler, rich into poor encoding, you lose something. And I was losing the right single quote.

Why is this so entertaining?

Because it’s everywhere, and because often what we think is going on is not what really is going on, and sometimes this can create problems. Haven’t you noticed how tricky it is to just copy and paste simple ASCII text these days without carrying markup and what not along with it? That alone requires a blog post. And if you paste a non printing character into a website database or a website file, that can create strange problems. The non-printable character is not seen and yet it breaks the layout, since the browser tries to work around it, closes the tag, and makes what it can of the mess.

Character encodings are everywhere. Web sites have them (check out View->character encoding in Firefox or Wrench menu->Tools->Encoding in Chrome). Databases have collations. Web applications too. And all these have to be aligned and also made to work well for different users. We have to keep track of encodings when moving between programs and operating systems. We need to know how much can we trust our text editors to not create problems. Etc.

A phrase is made of bytes, which are numbers. We see letters, digits, symbols, ideograms, and what not, but underneath it’s numbers, and encodings give them meaning. When we need to print or read letters and numbers and symbols, computer programs translate them into numbers and back again.

If it was simple ASCII, every character would just be represented by one 7-bit byte. This means that there are 128 (0-127) ASCII characters, printable and not, and the 128th character, decimal 127, has a binary value of 1111111. You are probably thinking that I miscalculated the bits, since they are 7, not 8 like in the byte we are used to. No, seven they are. Someone told me that in the teleprinter old days the 8th bit was used for parity, as a sort of a checksum, so you could check if the rest of it came through plausibly right. The idea was kept through the modem days, apparently (see XMODEM and ZMODEM file transfer protocols). But don’t worry, there is the extended ASCII (or high ASCII) that goes into the higher numbers and describes 8-bit or larger character encodings. Letters with accents are in the extended part, and some old school computer fanatics still refuse to have anything to do with them in e-mail and such and stick to the basics:) Oh yeah, there’s also the ASCII Ribbon Campaign against HTML in e-mails which makes more sense every day in this world of malicious Javascript code passed in img tags and such. Oh well.

And now go see some ASCII art to relax. A bit. Ha ha. Not funny 🙂

So, I found the weird 3 bytes (“e2 80 99“) in my string and Googled them. It turned out to be some Microsoft specific right single quote, that takes well 3 bytes to be represented in Unicode. UTF-8 Unicode encoding uses different numbers of bytes to represent characters – sometimes 1, sometimes 2, sometimes more. UCS-2 is fixed length, UTF-16 uses 1 or 2 bytes for a character, and so on. ISO and IEC have their series of standards for 8-bit character encodings. Microsoft has their own character sets.

In the Wikipedia page you can check out the web resources, including the character encoding Wikipedia page.

I could go on, but I’ll just admit I have to study much more yet, and some day maybe put together a tutorial on the subject.

In the meantime I’d like to remind you to learn something new every day, big or small makes no difference, and keep a diary of what you learn. It will help you see the day in a new light. Oh, BTW don’t compare yourself to others. Mostly they have a lot to learn, but hide it well.

Model Thinking – models as the new Lingua Franca – notes from new online course by Scott E. Page

I’ve brought up the concept of models before. From the MIT online courses, I learned that programming is about mapping the world into something computational. I’ve had to do with models as part of the model-view-controller pattern in programming, and when modeling databases into objects. I’ve blogged about Bruce Schneier and the importance he gives to models, explaining how feelings are our interpretations based on intuitions, and models are based on reasoning, be it our own, or inherited. Models are important, and determine us a great deal, and it all influences our predictions, our decision making, our lives, presents and futures. Models influence both how we perceive our lives and feel while living them, and what actually happens to us based on the decisions we make and things we do.

Clearly I am very happy to be following a whole online course in model thinking and see where this serendipitous intuition can lead me and what it can teach me. I have discovered it when new Stanford free courses were announced, and it is taught by Scott E. Page.

I have a background both in computers and in social sciences, and models kind of fill the gap between them, linking them in a way I always felt was there. I like computers for computers’ sake, but playing with them and not going beyond that has never been enough for me. At the same time, when I would study something from other fields, I would find myself applying methods that programming has made a part of me. Model thinking helps me make sense of that, maybe. Scott E.Page himself is not a Computer Scientist, but a Professor of Complex Systems, Political Science, and Economics at the University of Michigan.

Clearly, the map is not the territory. And in the first two video lectures of the Model Thinking course Scott E. Page shares this quote with us:

“Essentially, all models are wrong, but some are useful.”
George E.P. Box

Click “Continue reading” to learn something about what the models are good for.

Living inside Moodle e-learning platform, source code and db structure

I’ve spent the last months of workdays inside the source code and db structure of Moodle, an interesting e-learning platform. My first impression was confusion. Some of the code is procedural, some is object-oriented, some global variables are used, and the database is a forrest of related tables.  The illustration below gives an idea of that. But it is a worthwile experience. I’ve learned to read and write Moodle, to modify it and expand it with custom plugins, to debug it and figure out what is going on even when the log files aren’t helping at all.

By Martin Langhoff from http://moodle.org/mod/forum/discuss.php?d=3935

MESS VIEW CONTROLLER 🙂

Moodle has its upsides and downsides. Coming to it after the Model-View-Controller world of CodeIgniter based portals was a shock. It isn’t Model-View-Controller at all! I thought OMG everything is just all over the place. The logic was being mixed with presentation in ways that for me were absurd. I even searched the forums to understand more and found out that the Moodle coders don’t see changing this as a priority and aren’t interested in aggressive criticisms on this account (not that it ever came from me). So I just got used to it and moved on. Sometimes I still get irritated when I see how functions and classes and libraries seam put together to fit the purpose of the moment and not planned out for possible future uses. But I can’t be too harsh. Moodle was written for teachers, not for programmers, and it has made many valuable online courses possible and enriched learning experiences of so many people, and keeps to do so. So I humbly keep exploring it and being inspired by it, and hope to be able to contribute to it some day too. Probably soon.

In Moodle things happen in three basic places: the source code, the database and the moodledata folder. The database plays a huge role: it keeps it all together, “knows” all about the plugins and logs and users and their activities. The source code is complex and lego-like. The moodledata folder keeps files.

Click “Continue reading” for some detail.

The Code is your Tutor, RTFSC

In a perfect world every apprentice would have a perfect tutor.  All the time. An apprentice would always be able to learn from great masters and journeymen under favorable conditions, they would have time and communicate well and learn from one another. The apprentice would offer enthusiasm and freshness, and the teachers would bring perspective and knowledge and best practices and even wisdom.

In the real world it’s much more chaotic. Our paths sometimes cross, sometimes they don’t, sometimes we have time, sometimes just issues and deadlines, but we always have the code.

We always have the code, and the code will guide us.

http://abstrusegoose.com/strips/you_down_wit_OPC-yeah_you_know_me.png

The code has always taught me something. Bad code lets me learn from its errors, good code from its elegance and efficiency. I travel through its lines, explore its streets, squares, cities, towns. It sometimes feels like living inside the code, I can really see it all around me. When I started programming, I thought that reading other people’s source code would be the hard part, but it wasn’t. When it comes to full immersion, you just do it, and don’t look back. You just learn. You just swim. Not always perfectly, mistakes are inevitable, but you do.

And I have only just started to explore. I have many more open source projects to take part in, functions and methods to invoke, classes to create and subclass. I wish I had used Google Code Search more before they closed it (anyone know of anything similar?).

No matter what the future brings, one thing is certain : I will keep learning. And knowledge is something no one can take away from you.

Does this mean it is OK to treat people with a “Read The F* Source Code” and “It’s all in the code, don’t ask questions” attitude? No it’s not OK. Reading the code is no substitute for communication, commenting or documentation.

But at least I know I will always have a great teacher, with an infinity of things to teach me, and that is the source code of all the free and open software out there. Thank you all for sharing.

[LINUX NOTES] Take your shell with you everywhere – we all SCREEN for I SCREEN

This week I’ll improve my usage of screen and look up byobu too. Here are some reasons why.

TABS IN THE COMMAND LINE

Modern browsers have used us to the concept of tabs. Interactive web sites are full of tabs. Why not have them also in the shell? With screen and byobu you can.

BETTER SCRIPTING PRODUCTIVITY WITH TABS

You can write your script in one tab and execute it in another after every modification! And in tab 0, for example, you can keep the tail -f of the apache error log when you are developing web applications. You can keep different logs in different tabs and monitor stuff all over the place, even on other machines, as you go on.

MEMORIZE FEW COMMANDS OR CREATE YOUR OWN

There aren’t that many command to learn, and you can also bind your own keystrokes to actions in the screenrc configuration file.

SHELL THAT DOESN’T CHEAT, IT KEEPS WORKING EVEN IF YOU’RE NOT LOOKING

So, the shell continues working for you even if you’re not looking. You can dettach the screen on any remote machine as you leave the office and come back home, reconnect to the machine, and continue working from where you left off. You can leave a tail -f that scrolls files you want to keep track on. Or use watch (-d for colors that indicate changes, -n or –interval for interval).

IF YOUR HAVING CONNECTIVITY PROBLEMS

If you are having connectivity problems, broken pipes, ssh shells freezing on you, you certainly need screen, along with other measures. BTW look up nohup as well 🙂

ADVANCED MONITORING

“Just wanted to mention to other cool tricks you can do with screen. Screen can monitor a window for activity or lack thereof. This is great if you are downloading large files, compiling, or watching for output. If you are downloading something or compiling, you can watch for silence. To start the monitor, go to the screen you want to monitor and use “Ctrl-A” “M” to look for activity or “Ctrl-A” “_” to monitor for silence. Then open or switch to a new window. When the monitor detects activity or silence, you will get an alert at the bottom with the window number. To quickly go to that window, use “Ctrl-A” “ (thats a quote mark, ctrl-a then a “). After you do this, just type in the number of the window and enter. To stop monitoring, go to that window and undo the monitor with the same command. For example, to stop monitoring for activity you would use “Ctrl-A” “M” again.”

http://www.rackaid.com/resources/linux-screen-tutorial-and-how-to/

INVITE USERS INTO A SCREEN SESSION

An interesting teaching tool if you want to show someone how something is done in the command line. I still have to try that!

SCROLLING AND SEARCHING

You can’t do the usual shift and page up to scroll like in a normal shell, when in screen. But you can

http://www.samsarin.com/blog/2007/03/11/gnu-screen-working-with-the-scrollback-buffer/

CUSTOMIZE SCREEN WITH SCREENRC

“A typical application of this tidbit of knowledge is to launch a predefined set of windows at screen startup. Below is a sample screenrc that will do so:

# read in your normal screenrc
# before anything else
source $HOME/.screenrc
# now start opening windows
screen top
# it's possible to set the window title with
# the -t option
screen -t irc epic
# you can also specify the window number
# to launch in
screen -t mail 8 mutt
screen -t daemon 9 tail -f /var/log/daemon.log

If you save this to $HOME/.screenrc.multiwin you can tell screen to use it instead of your normal .screenrc by running screen -c $HOME/.screenrc.multiwin”

http://www.linuxjournal.com/article/6340?page=0,1

I’ll write a separate post on that some day 🙂

More links:

– http://www.kuro5hin.org/story/2004/3/9/16838/14935

– http://www4.informatik.uni-erlangen.de/~jnweiger/screen-faq.html

– http://www.softpanorama.org/Utilities/Screen/screenrc_examples.shtml

Happy Mondays, or why we need to reboot regularly

I’ve had some long weekends and holidays lately and had to start over on Monday, or Tuesday which is not all that different, like today. I was surprised to find out that it is not as bad as I thought. I’d go to sleep Sunday worrying about the shock awaiting me the next day, thinking I’d fall asleep at the computer, or have a hard time getting the hang of things. Fortunately, it was much easier. As I got on the train for my daily commute I was already changing my mood, and as I entered the office and turned on the PC, it was truly a new day and it felt like one. I saw things in a new light and had a new energy to deal with them. Fresh ideas came to me, and I was better at switching points of view and being creative. And relaxed. And focused.

But not all Mondays are like that. Maybe I need to rest enough on weekends to have a Monday like that? Who knows. I sure have to figure out the trick of it.

The New Year is also a big mega Monday. I’m starting to get all these New Year resolution ideas and have to remind myself not to exaggerate with expectations of myself. This is going to be a year of balance, then! No more torturing myself with perfectionism. I’ll learn to allow myself to see myself and the world from my own point of view, and live my life as such – mine.

Click for more musings on the subject 🙂 and have a great New Year full of Great Happy Mondays!

2011 in review, happy 2012 to all!

What a year! I’ve returned to programming, learned so much and found a new job! I entered the world of e-learning platforms both as student and as coder, I’ve meet Drupal and deepened my knowledge of Code Igniter,  I extended my object oriented construction and patterns skills, practiced some SQL, even tried taking courses in machine learning and AI!

I’ve figured out a lot about what to do and what not to do, about managing my energies and nerves, preserving my enthusiasm through it all – after all, this is a marathon, not a sprint!

Now a new year starts, and I’m a Junior Developer. Let’s hope it all goes well. I’ll keep you posted. Yes, I haven’t blogged lately, but that’s about to change.

Happy 2012 to you all!

BTW, the WordPress.com stats helper monkeys prepared a 2011 annual report for my blog.

Here’s an excerpt:

A New York City subway train holds 1,200 people. This blog was viewed about 7,200 times in 2011. If it were a NYC subway train, it would take about 6 trips to carry that many people.

Click here to see the complete report.

Beyond good enough, the more you study for the easy parts the better you’ll be at what’s difficult

Ok, I’m back. It’s been a busy few months, and I’ll likely explain why in another post. I’m a busy PHP Junior Developer now, and I’m also taking 3 online courses (offered in partnership with the Stanford University), as you’ve read in the previous posts. I’m following Introduction to Databases, Machine Learning, Artificial Intelligence. And, obviously, I’m having a hard time balancing it all: especially the time, the energy and the motivation. I’ve just taken my DB midterm quiz, and am not too happy with the results. I’m ok with the AI homework, if I manage to submit the one that is due tomorrow. And I’m behind with the Machine Learning programming exercises.

I’m not starring the courses. But that’s ok, this is just a test anyway, to see if I can both work and study. I must learn some of the topics, but mostly I must learn from my mistakes and learn how to learn.

I have, clearly, made mistakes. But the important part is, what have I learned from them? Here’s a top 5 that answers that. Click “Continue reading” for the details.

5) Plan ahead. Plan the rest days as well. Then live each day to the fullest.

4) Add some aesthetics and some fun. Take great notes. Detailed great notes.

3) Don’t consider a course easy if a part of it is easy, or difficult if a part of it is difficult.

2) 80% of the work may be done with 20% effort, but A students do more than that.

1) The more effort you put into the easy stuff, the better you will be on the trickier parts.

First steps in Machine Learning

My e-learning has started, a week before the official start of the Stanford online classes. They have already made available some videos and quizzes, and us students have started strong. The Machine Learning class Twitter account announces some interesting numbers, talking about tens of thousands of students enrolled and quizzes attempted.

I have already started learning the basics of Machine Learning, which starts with the definition of Machine Learning itself. At a certain point, it has been noted that there are some problems for which it is better to let the computer learn by itself, instead of programming it explicitly. That’s what the 1959 definition by Arthur Samuel is about: it states that ML is the field of study that gives computers the ability to learn without being explicitly programmed. This may sound strange, since we know that computers can’t really do anything that we haven’t programmed them to do, or some variation of that. But think about this: Arthur Samuel was not a great checkers player, but he managed to teach the computer to play and improve with every time, until the computer became better at checkers than he was. Also when scientists wanted to teach a helicopter to fly autonomously, they found that the best thing was to let it learn on its own. But how does that happen?

The same way that we do, computers sometimes learn from experience. They repeat and learn, repeat and learn. That brings us to the second definition of Machine Learning: it allows computers to improve its performance over time on a specific task, from experience. It was coined by Tom Mitchell in 1998 and the complete phrasing is that a computer is said to learn from experience E with respect to some task T and some performance measure P if its performance on T, as measured by P, improves with experience E.

What does that mean concretely? One way is that basically we can give a computer a training set of data as input, and it can try to find a function which fits the data reducing the amount of errors, it can basically infer a function to  use to predict further similar results (like apartment prices given a set of variables). And I’ve already taken a sneak peek at the mathematics behind that.

Which is obviously extremely exciting and I would like to thank Stanford, Prof. Andrew Ng and the whole ml-class.org team for this opportunity.

Click “Continue reading” for more details.

[EDIT] You can find some Machine Learning video lectures by Andrew Ng at http://openclassroom.stanford.edu/MainFolder/CoursePage.php?course=MachineLearning