SSH Key Auth Fails when using Git with Sudo

Flashback about two years, and I had never touched git. GitHub was that place where you clicked the Download link to get a zip of the code you wanted. That being the case, I am still learning as I go. The other day, I drove myself crazy over a complete n00b mistake, which I am embarrassed to admit.

While working on an Ubuntu server, I was trying to pull changes from a Bitbucket repo into a subdirectory of /var/www. Every time I ran “git pull origin master,” the following error was displayed:

Permission denied (publickey).
fatal: Could not read from remote repository.

I knew the correct SSH key had been added to my profile, but I ran “ssh -T hg@bitbucket.org” to be sure. This returned the positive and expected:

logged in as jdoe.

So, why did it fail when I tried a git pull? I double-checked my remote, removed and re-added it to be sure. Then, it hit me! Because of the permissions set on the current directory, I was having to use sudo for all my git commands. But, sudo doesn’t know about my SSH keys (at least by default). After realizing this, the fix was easy: correct the permissions on the directory (which I should have done from the beginning instead of working around it with sudo) and add myself to the directory’s group. Sudo was no longer required and my authentication attempt worked beautifully.

Again, I hang my head in shame over this, but thought I should share in case someone else has a temporary lapse in judgement.

(While verifying the cause, I came across a quick guide that will forward your key to sudo, if you really must.)

Crawling Commented Styling with Heritrix

This post isn’t something I can take credit for. The purpose is to make two potential solutions discoverable for someone like me, looking for an answer. Credit will be given where it is absolutely due.

As I’ve written about before, I inherited a Wayback/Heritrix server in my role and have had the pleasure of hacking my way through it on occasion. A recent challenge arose when I needed Heritrix to crawl an old Plone site. The Plone template placed html comments around all the styling tags to hide it from older browsers, which couldn’t understand the CSS. The result looks something like this:

<style type="text/css"><!--
/* - base.css - */
@media screen {
/* http://servername.domain.net/portal_css/base.css?original=1 */
/* */
/* */
.documentContent ul {
list-style-image: url(http://servername.domain.net/bullet.gif);
list-style-type: square;
margin: 0.5em 0 0 1.5em;
}
--></style>

Unfortunately, it seems Heritrix happily skips past any URLs within comments by default and does not follow them, regardless of your seeds and other configurations. Because, hey, they’re only comments, right? The end result is that it looks like the site was crawled successfully, but some resources were actually missed. In the above example, the Wayback version of the site was still pointing to http://servername.domain.net/bullet.gif for thelist-style-image, rather than http://wayback.domain.net:3366/wayback/20151002173414im_/http://servername.domain.net/bullet.gif. Therefore, it was not a complete archive of the site and its contents.

In my case, this was an internal site that I had total control over. However, try as I might, I could not figure out how to remove the comments from the old Plone template. Grepping for ‘<style type=”text/css”><!–‘ turned up ‘_SkeletonPage.py’. I tried modifying it and then running buildout to no avail. I am sure people more experienced with Plone could tell you where to change this in a heartbeat, but it’s beyond my knowledge with the application at this point. After coming up short on searches for solutions with Heritrix (thus, this post), I started looking for ways to remove the comment tags with something like Apache’s mod_substitute, since Plone was being reverse-proxied through Apache anyway.

Solution 1: Mod_Substitute/Mod_Filter

Eventually, I stumbled upon this configuration from Chris, regarding mod_substitute and mod_filter. Mod_filter needed to be used for mod_substitute to work properly because of the content being reverse-proxied. A simple modification of Chris’s configuration worked to remove the comment tags beautifully (using CentOS/Httpd for reference):

LoadModule substitute_module modules/mod_substitute.so
LoadModule filter_module modules/mod_filter.so
FilterDeclare replace
FilterProvider replace SUBSTITUTE Content-Type $text/html
FilterChain +replace
FilterTrace replace 1
Substitute "s/css\"><!--/css\">/n"
Substitute "s|--></style>|</style>|n"

(Note: I probably could have made this a little more efficient by using a single regex instead of two separate substitutes. But, meh. This was good enough.)

Chris recommended loading this into a new file: /etc/httpd/conf.d/replace.conf.

Solution 2: Hack Heritrix

While exploring my options with Apache, I also decided to reach out to the archive-crawler community on Yahoo! for help. A user identified as “eleklr” shared a patch that he used often for this kind of scenario. I think this is the better route to go, though I have not had an opportunity to try it out yet. Its biggest strength is that it doesn’t require you to have complete control over the site you are crawling, as is necessary for solution 1.

If you’ve found yourself in my position, rejoice in the fact that it’s not just you and there are solutions out there. Hopefully, one of the two listed above will help you on your way. Please share if you’ve discovered other solutions to this or similar problems.

Using historical dates in WordPress custom fields

I’ve been working on a family history project in WordPress, which includes a bit of a genealogy. Naturally, no genealogy is complete without important dates, such as birth and death. However, as I moved further back in history, I ran into a particular annoyance that seems to bother many WordPress developers (http://wordpress.stackexchange.com/questions/158663/permitting-wordpress-to-accept-dates-outside-of-1902-2038).

A handful of custom date fields were created for this project. They worked well, until I attempted to add an ancestor born in the late 1700’s. The furthest back the date picker allowed was 1902. No bother, I thought, and simply typed the date into the expected format. But, when I clicked Save, the date fields were emptied. Some googling indicated that it was a problem with 32-bit PHP not supporting a large enough negative integer for the strtotime() function to represent a date earlier than 1902. This question on StackExchange has two great answers that include more detail. The problem was, I have 64-bit Apache (which is determined by the PHP constant PHP_INT_SIZE equaling 8).


You might wonder why PHP’s strtotime() function would prevent certain dates from inserting into a date field in MySQL. The important thing to realize is that every custom field’s value is stored as longtext in the wp_postmeta table and not a date field. So, you aren’t saving a date; you are saving a string, that represents and is later displayed as a date.


 

Considering that PHP’s date/time functions rely on the underlying OS’s date/time functions and the server was running Ubuntu 12.04 (over two years old now), I decided to try an upgrade to Ubuntu 14.04. By the way, here’s a big tip: Upgrade your Linux distro before it reaches End Of Life. Waiting until after makes it a bit of a headache. Anyway, after the upgrade completed, I went back to WordPress and saw that the field was still limited to 1902. So, I updated WordPress to 4.0. That did the trick! I can now go way way way way way back in time. If I studied the problem and release notes of each upgrade more thoroughly, I could probably tell you why, but upgrading Ubuntu and WordPress took care of the problem for me in any case. Just another example of why staying up-to-date is always important.

If someone a bit more versed in WP/Ubuntu changes cares to share further insight, please consider sharing in the comments.

Install mod_proxy_html in CentOS

Just a quick note on installing mod_proxy_html for Apache in CentOS. I have seen several methods, but, assuming you have the EPEL repos added to Yum, it is pretty much a one-liner:

yum install mod_proxy_html

Once installed, it is just a matter of adding the following line to your httpd.conf file and restarting Apache:

LoadModule proxy_html_module modules/mod_proxy_html.so

This will allow you to use the ProxyHTMLURLMap directive. To learn more, check out Apachetutor.org’s tutorial.

jQuery getJSON from PowerShell via PHP on IIS: A Frustrating Gotcha

(Once you have your PHP on IIS environment setup and ready to go, you can check out this example code. You can also take a look at my new project, Gumper, on Bitbucket – an API for executing PowerShell and other scripts remotely.)

And in 40 years, this subject line will make absolutely no sense whatsoever…

About two years ago, I wrote a PowerShell script that generated a .Net form with all sorts of tools for our Help Desk. By searching for an Active Directory user, the Help Desk could instantly see if the user’s account or password was expired, whether or not the user had a mailbox, if the user’s mailbox database was mounted, quota information, and more. Recently, I revisited the script and decided it was time to take it to the web. Running these tasks from a PowerShell GUI is not terribly flexible or efficient. So, I set out to learn what it seems many Sys Admins like myself are learning: how to run PowerShell scripts via PHP.

Right out of the gate, I am already committing a bit of heresy. Rather than installing WAMP, I decided to stick with the IIS instance where I had already installed PHP (http://blogs.iis.net/bills/archive/2006/09/19/How-to-install-PHP-on-IIS7-_2800_RC1_2900_.aspx). And why use PHP to run Microsoft PowerShell (built on .NET)? Quite frankly, I like PHP and find it easy to learn. This project is still in its infancy, but there have already been a few important snags I thought worth sharing.

Use shell_exec()

If you want to launch a PowerShell script from a PHP file, you have a few options: exec(), shell_exec(), and system(). Using system() doesn’t seem to make sense if you intend to get a response from the server. This would be intended more for something like kicking off a scheduled task or another background process. Exec() will do the job, but it will split your response up into an array based on line breaks. This might be OK depending on how you want data returned. But, for my purposes, I chose shell_exec() so I could format the data a bit. Shell_exec() will return the output of your script as a string.

Keep your script silent

Note that “shell_exec()” returns all output from your script. That means errors, write-hosts, and everything else that pops up when you run a script. So, be sure to make your script as silent as possible and only “return” the little bit of data you want passed into PHP. This might mean a lot of Try/Catches (which is a good practice anyway).

Launch your script

This was actually a pretty easy one to conquer. Many people have examples of the basic syntax to use in order to launch a PowerShell script. Here is an example of what I am using:

shell_exec(“powershell -NoProfile -File $scriptPath $argString < NUL”)

(You may also need to pass the “-ExecutionPolicy” switch with a value depending on your setup.)

The most critical part of this is the “< NUL” bit at the end. Without it, PHP will never get the output from PowerShell and will, instead, wait and wait and wait. You will also notice that I use two variables: $scriptPath and $argString. These are PHP variables that I pass into a function and are used to call the script file along with any arguments. So, if $scriptPath is “C:\web\script1.ps1” and $argString is “-User jdoe”, the above line would render as:

shell_exec(“powershell -NoProfile -File C:\web\script1.ps1 -User jdoe < NUL”)

Authenticate

Remember that IIS has different options for authentication. I chose Basic with HTTPS, but someone else may have a better idea since I really just went with whatever worked first. The main thing is to turn off Anonymous authentication. The reason Basic works well for my situation is that each page runs as the authenticated user and their permissions. The importance of this will become evident below.

Enable fastcgi.impersonate

Even if you are authenticated, you probably won’t be able to launch any PowerShell scripts by default. This is because PHP does not pass along your authentication to the Windows command line unless fastcgi.impersonate is enabled. Enabling this in php.ini makes it so every PHP script that runs, runs as the authenticated user. Keep that in mind, because it may change how you design your site.

To enable fastcgi.impersonate, locate your PHP.ini file (Maybe C:\Program Files (x86)\PHP\v5.x\php.ini) and un-comment the line that says “;fastcgi.impersonate = 1;” by removing the semicolon (;) at the beginning of the line. The line should look like this when you are done:

fastcgi.impersonate = 1;

After this, save php.ini and restart your web site.

Return JSON

If you are going to manipulate the data inside the browser, it makes sense to return your data from PowerShell as JSON. This is pretty straightforward. For example:


$a = "apple"

$b = "banana"

$c = "coconut"

$json = "{`"a`": `"$a`",`"b`": `"$b`",`"c`": `"$c`"}"

Return $json

This should spit out the following:


{"a": "apple", "b": "banana", "c": "coconut"}

Be sure to wrap string in escaped quotations (`”) to keep your JSON valid. And validate your syntax with JSONLint (http://jsonlint.com/).

Beware the Console Width

This is the real reason for writing; the thing that nearly made me go brain-dead. I chose to use jQuery’s getJSON function since I was already returning a JSON array. There are many great tutorials that show you how to accomplish this, so I won’t get into that. Despite all the great tutorials, I was getting nowhere. No matter what I did–callback to a function, change the mode to synchronous (WHAT!?), use $.ajax, click with my right pinky-finger–nothing worked. I could see the data in FireBug, but I could not get anything to show up on the page. This frustrated me all night. Today, I was watching the same data pass through FireBug, refreshing, clicking again, and feeling utterly hopeless (OK, so maybe there are more important things in life than PowerShell and PHP), when I finally realized something important: maybe FireBug wasn’t wrapping that really long Distinguished Name in the JSON array for easy reading. Maybe the data was coming back to the browser with a real line break, thereby invalidating the JSON.

Yup, that was it.

It turns out, when shell_exec() launches PowerShell, the data returned is formatted to the default console size: 80 characters wide. Meaning, if your JSON object goes beyond 80 characters, it will break to the next line of the console and the data you get back will be invalid. For example:


{"distinguishedName": "OU=weeny,OU=teeny,OU=bitsy,OU=itsy,DC=reallylongnamiccusd

omainiccus,DC=net"}

Instead of…


{"distinguishedName": "OU=weeny,OU=teeny,OU=bitsy,OU=itsy,DC=reallylongnamiccusdomainiccus,DC=net"}

So, what are the options? Well, there are two I can think of. First, build well-formed line breaks into your JSON like so:


$a = "apple"

$b = "banana"

$c = "coconut"

$json = "{`n`"a`": `"$a`",`n`"b`": `"$b`",`n`"c`": `"$c`"`n}"

Return $json

Adding the new line character (`n) will cause the JSON to break in a spot that will not invalidate the array. This should return:


{

"a": "apple",

"b": "banana",

"c": "coconut"

}

Option 2 is to adjust the console size if your script will output more than 80 characters on a line. The “Hey, Scripting Guy!” blog has a great article on how to do this: http://blogs.technet.com/b/heyscriptingguy/archive/2006/12/04/how-can-i-expand-the-width-of-the-windows-powershell-console.aspx

As I said earlier, this project is in its infancy and I am sure many more gotchas await me. However, this last one seemed like something others might run into. It’s so silly, but so frustrating. And, I imagine anything run from the command prompt will yield the same results.

Happy coding.

Adding Email Functionality to Quizzin (Plugin for WordPress)

This post is well overdue.

Quite a few months ago I had the pleasure of developing a web site for a textbook. One of the desired features was the ability to take quizzes and receive instant feedback on these quizzes. We were working with WordPress, so I thought finding a plugin to do this would be quite simple. I was mistaken.

After trying out several quiz/survey plugins, Quizzin by Binny VA (http://projects.binnyva.com/languages/php/quizzin-wordpress-plugin-quiz-creation-plugin/) was by far the closest to what we were looking for. We could deploy quizzes and include feedback for correct and incorrect responses. Perfect. Except for one small thing, there was no way to add the option to email the student and/or instructor with the results. If you have ever taken a quiz on a textbook’s web site, you will understand how important this feature is.

I contacted the developer of Quizzin and requested that he add the feature. He was kind enough to respond quickly letting me know it should be pretty easy to accomplish. Bless his heart. That’s when I took a crash course in PHP mail. I started examining every line of code in the plugin and trying all sorts of combinations, coming frustratingly close. After exhausting my wife’s patience, I contacted the developer again. I sent him all the code I had worked out and asked for assistance. His response? Move the code up two lines. Genius! He changed my WordPress life in that moment and steered me toward a working quiz tool.

So, if you are one of those desperate developers attempting to accomplish this same thing, I recommend you install Quizzin (I used version 1.01.4) and make these modifications (or something that fits your needs):

  • Open the “show_quiz.php” file within the Quizzin plugin
  • Locate this code…
    <?php if($answer_display == 2) { ?>
    <input type=”button” id=”show-answer” value=”<?php e(“Show Answer”) ?>”  /><br />
    <?php } else { ?>
    <input type=”button” id=”next-question” value=”<?php e(“Next”) ?> &gt;”  /><br />
    <?php } ?>
  • Create a form with code like this on the very next line after the <?php }?> tag listed in my previous bullet point…
    <strong>Before clicking “Show Results”</strong>: Please enter your instructor’s email address, your email address, and your name. Results from this quiz will be sent to all email addresses entered below once you click “Show Results.”.<br /><br />

    <table border=”0″>
    <tbody>

    <tr>

    <td valign=”top” width=”125″> Instructor’s Email: </td>
    <td valign=”top” width=”*”> <input type=”text” name=”instr_email” size=”18″ /></td>
    <td width=”300″></td>

    </tr>

    <td valign=”top”>Your Email: </td>
    <td valign=”top”><input type=”text” name=”stu_email” size=”18″ /></td>
    <td>(separate email addresses with commas to add multiple recipients)</td>

    </tr>

    <td valign=”top”>Your Name: </td>
    <td valign=”top”><input type=”text” name=”stu_name” size=”18″ /></td>
    <td></td>

    </tr>

    </tbody>
    </table>

    <br />

    <input type=”submit” name=”action” id=”action-button” value=”<?php e(“Show Results”) ?>”  />
    <input type=”hidden” name=”quiz_id” value=”<?php echo  $quiz_id ?>” />
    </form>

  • In the same file, locate this code…
    // Show the results

    print str_replace($replace_these, $with_these, stripslashes($quiz_details->final_screen));
    if($answer_display == 1) print ‘<hr />’ . $result;

  • Insert something like the following code on the very next line after the “if($answer_display == 1) print ‘<hr />’ . $result;” line in my previous bullet point, but remember to customize it for your use…
    //Send an email with results

    function make_safe($variable) { $variable = mysql_real_escape_string(trim($variable)); return $variable; }

    $to = make_safe($_POST[‘instr_email’] . “,” . $_POST[‘stu_email’]);
    $stu_name = make_safe($_POST[‘stu_name’]);
    $headers  = ‘MIME-Version: 1.0’ . “\r\n”;
    $headers .= ‘Content-type: text/html; charset=iso-8859-1’ . “\r\n”;
    $headers .= ‘From: My Web Site <admin@mywebsite.com>’ . “\r\n”;
    $headers .= ‘Reply-To: admin@mywebsite.com’ . “\r\n” .
    $subject = ‘Quiz Results for ‘ . $stu_name . ‘, ‘ . $quiz_details->name;
    $email_results = ‘<font color=”#000000″><br />Please do not respond to this email<br /><font size=”5″>Quiz: ‘ . $quiz_details->name . ‘<br />Student Name: ‘ . $stu_name . ‘<br />Score: ‘ . $score . ‘ of ‘ . $total . ‘<br /></font>’ . $result . ‘</font>’;
    mail($to,$subject,$email_results,$headers);

  • Save your changes and you should be all set (unless I am forgetting something since I did this some time ago now)
  • Make a back-up of the Quizzin plugin since any updates will overwrite these custom changes.

There were a few styling changes I made so it would be more obvious in an email whether a question was answered correctly or not. I changed the if(strpos($class,’answer correct-answer’)) $result line to if(strpos($class,’answer correct-answer’)) $result .= “<font color=’green’> (Your selected answer is correct)</font>”; and if(strpos($class,’ user-answer’)) if(!strpos($class,’ correct-answer’)) $result to if(strpos($class,’ user-answer’)) if(!strpos($class,’ correct-answer’)) $result .= “<font color=’red’> (Your selected answer is incorrect)</font>”;.

Though CSS is generally the way you want to go with styling, remember that your CSS file will not be present in the email received by the student and/or instructor. This is why I used the font tags. Good luck and may some poor soul have an easier time of it than I did.

listMedia: an HTML5/JavaScript Function

I attended a brief workshop in Grand Rapids yesterday with Phil Gerbyshak. He talked about sharing one idea with someone everyday, which is why I feel motivated to share this bit of JavaScript with the Interwebs.

As I was going over some HTML5 tutorials, I became excited about the video tag. True, it does not add anything that could not be accomplished with HTML4, but it does it in a much cleaner way. Perhaps my favorite part is the built-in intelligence with recognizing supported media types. Say you aren’t sure whether an OGG or MP4 is better to use for compatibility sake. HTML5 says, “why not use them both?” By using the following code, the visitor’s browser will automatically select the very first file type it recognizes:

<video width="320" height="240" controls="controls">
  <source src="movie1.mp4" type="video/mp4" />
  <source src="movie1.ogg" type="video/ogg" />
</video>

If the browser does not recognize movie1.mp4, it will simply move onto movie1.ogg. And for the record, “controls=’controls'” and “source src=” have to be some of the most redundant HTML tags I have seen in my life.

This is all great, but it got me thinking. Even though there are only three primary video formats (kinda) supported in HTML5 (OGG, MP4, and WebM), what if this list grows? Five years form now, we don’t want to write out fifteen lines of possible video sources “just in case.” So, I set out to create a JavaScript that would do this for us. I call it listMedia and it currently has a strict and loose version.

listMedia Strict

This code would go into your <head> or an external JS file:

<script type="text/JavaScript">
function listMedia(mediaAddr, mediaType)
{
if (mediaType="video")
   {
   var vidFormats= new Array("ogg","mp4","webm");
   for(var i=0;i<vidFormats.length;i++)
      {
      document.write("<source src='" + mediaAddr + "." + vidFormats[i]
      + "' type='video/" + vidFormats[i] + "' />");
      }
   }
else if (mediaType="audio") //checks to see if the media type is audio
   {
   var audFormats= new Array("ogg","mp3","wav");
   for(var i=0;i<audFormats.length;i++)
      {
      document.write("<source src='" + mediaAddr + "."
      + audFormats[i] + "' type='audio/" + audFormats[i] + "' />");
      }
   }
else //media type failed to be detected
   {
   document.write("<source src=
   'http://myweb.arbor.edu/jthiede/no-type.mp4' />");
   }
}
</script>

Then, you would place something like this in your <body>:

<video width="320" height="240" controls="controls">
   <script type="text/JavaScript">
      listMedia('http://www.w3schools.com/html5/movie', 'video');
   </script>
Your browser does not support the video tag.
</video>

What’s happening?

Starting form the bottom and going up… A bit of JavaScript nested in the <video> tag calls the “listMedia” function and passes it two variables: most of a URL (http://www.w3schools.com/html5/movie) and the type of media (video). These variables get processed by listMedia and checked to see if the media is a video or audio file. Since the mediaType is set as “video,” the variables go through the first IF condition.

An array called vidFormats is used to keep up with all the currently supported video formats. Just a note – this array can be added to and stripped down to your heart’s delight without messing anything up. Finally, listMedia prints out a list throwing the appropriate extensions on the end of the mediaAddr variable. Voila! No need to type out every format.

listMedia Loose (aka Version 3)

You may notice that there is a perfectly useless “else” condition in the strict version. What was supposed to happen was that a video saying “please ask the web admin to select the proper media type” would show up if the media type was not selected or was invalid. However, while debugging this, I learned that it was not required to specify whether the media was audio or video, so long as your <audio> and <video> tags were used correctly. I was amazed to find that HTML5 allows a browser to check a media file against any extension, video or audio. Meaning, you can keep one array with all the media extensions, and your visitor’s browser will find the right one.

Thus, I created a simplified version of listMedia. My purpose for retaining both versions is that I do not know how strict HTML5 will become and I grow weary of bright red boxes on validators for not having something like an <alt> tag. For now, the loose version works great. However, this may change:

<script type="text/JavaScript">
function listMedia(mediaAddr)
{
   var mediaFormats= new Array("ogg","mp3","mp4","wav","webm");
   for(var i=0;i<mediaFormats.length;i++)
      {
      document.write("<source src='" + mediaAddr + "." + mediaFormats[i] + "' />");
      }
}
</script>

<audio controls="controls">
   <script type="text/JavaScript">
      listMedia('http://www.w3schools.com/html5/song');
   </script>
Your browser does not support the audio element.
</audio>

Smaller and more efficient. This is basically the same process, but we no longer care if the media is audio or video. Instead, we use one array to define ALL of the media extensions. The key for getting both versions to work is remembering not to pass the full path of your file to the mediaAddr variable. Instead of typing “http://mysite.com/video1.ogg&#8221; just type “http://mysite.com/video1&#8221;. listMedia will take care of the rest.

That’s really it for now. Honestly, so long as their are only five supported media formats, this may not be a huge time saver. But, it could make a big difference in the future. My next two small-scale projects will be using listMedia with CSS3 and porting listMedia over to PHP (which seem much more applicable IMHO).

Please feel free to offer any suggestions. To see listMedia in action, go to the terribly unflattering http://myweb.arbor.edu/jthiede/listMedia.htm. View the source of the page to see all the comments that I could not get to fit in WordPress.

Enjoy!