im@s: One for All Compendium and requests - thoughts and considerations.

[Setsuna]

I'm just going to leave this here, and have people consider this one way or the other.

This is what I've been thinking. You're free to debate what I've been thinking about it. All of the below is one reason I've been thinking about im@s request filling as a whole.

I've been considering if I should really do work towards the One for All Compendium and consequently the request service.

I know there'll be a significant request demand, mostly because of the fact that there's several features that weren't present in im@s 2, and I have a list of songs which will be present. (More contact work, I guess)

At the same time though...

a) The amount of hours I have to devote to this is quite immense. The fact the song only goes for about 2 minutes and 15 seconds is actually a very small part of the process.

Each video to record at high quality for me takes roughly an hour, end to end. (Half of that time is uploading said video, if you're curious. The other half is encoding it at very high quality.)

There's also the massive cataloging, the image preparation and other points. A lot of it can be reused from the im@s 2 compendium, but there's a LOT of work regardless.

To get it where the im@s compendium is at the moment took somewhere in the vicinity of about 70 hours for the base and for tweaks and updates, and at least 15 hours per catalog.

This includes testing time, determining shots, and a lot of other things.

If we assume that the process can be sped up for One for All, it reduces the coding part, but not the manual screen capturing and cropping process. So I'd reduce the hours required for the base to maybe 40, but the time for each catalog update to be the same.

It's actually a lot of work. It's also not a lot of fun, since I can name a few people who got burnt out helping me.

To be very honest, there's a cost premium on the time used, which I could be using towards writing in particular.

b) The amount of resources required to do requests is also immense. My net connection isn't usable during this period, and for the upload I get, I have to pay 100 AU a month to cover the costs. Encoding costs are done on a second machine, and this machine can't be used for other work during the encoding process as well.

Once again, there's a lot of opportunity and real costs (electricity comes to mind too, and I pay all my own bills) involved in offering the service.


c) It's bloody expensive to keep the compendium up to date. The compendium requires all the catalogs AND all their items to be purchased, to make them complete due to how Namco Bandai have this habit of hiding away items only after you buy EVERYTHING.

I'm tempted to run a check how many items in the catalogs for im@s 2 have been asked for, but I would hazard a guess and say roughly 70-80% of all the items in im@s 2 have shown up at least ONCE for a request.

If you're wondering, it cost roughly 1500 AU to get the game, and to make sure that the catalogs are available as they are released. It's not cheap.

Since I've got a significant cost to bear as it is (since I am now once again living on my own) I'm wondering if I can straight up afford to do this project.

d) There's been several personal circumstances that are going to significantly hinder the project both time wise and cost wise. Having half a face hurting like all hell is expensive to fix, among other things. I get to see a dentist, and that's going to cost me about a thousand all up, and I'm still paying moving costs from the move I made in late January.


The Bottom line: I can't afford to do it so easily compared to im@s 2, and it's taxing to constantly do it without any sort of ability to offset the costs.

Donating doesn't really help. I can certify I saw three donations the entire time during the im@s 2 run. I can name the people if I wanted to.


After the sheer amount of hours that I and a few others, like Crisu, who is also paying for webspace AND has provided significant amounts of coding hours to make it work, and several volunteers who spent time cropping images have put in, it's just... well, been used and that's how it is.

I also know given the amount of time required, that it'd be unlikely that we'll see that many people step up if no one else does it.

As for requesting, I know from experience that it gets piecemeal when it comes to requesting for accessories and the like, just due to the request stats. It makes me wonder if we'd see less variety in the videos being put out.


So what do I do? I don't know.

I'd argue that for at least im@s: One for All that at least minimum wage should be paid for labour costs, which in turn will get moved to the im@s: One for All fund to pay for catalog purchases.

I did the calculations, and assuming we take a paltry 5 minutes per video requested (It takes a lot longer than that, but we'll assume that the other costs are magically free), it amounts to roughly 2USD per video. (Minimum wage here is about 16 dollars an hour, 12 videos, then dock paypal fees.)

We can make arguments discounts for blocks of requests, but that's the math. Maybe those who help cover compendium costs can make requests in a priority queue, or something.

The other thing could simply be to restrict requests to the catalogs available, and they'll only be updated when funds become available, and limit maximum requests to a maximum of X amount a week.

Giving my current funding ability, and the history of im@s 2, I'm thinking we'll see the first catalog of im@s: One for All done... sometime in three years.

I'm certain that there'll be others to fill the gap, and if I'm REALLY lucky, someone else with a lot more resources than I currently have will just be able to take over the entire project as is, and I can just be a part time request filling service, dropping stuff here and there.

Do I really want to make a profit? Ha, considering how straight up expensive it is, I don't think I could possibly do it.

Would it be nice to see the community actually OWN the project and have it be more than one or two people's problems to bear? That'd be nice.

I spoke to a couple of friends who asked me about if I'd do a One for All Compendium, and they don't think that I'll get anywhere with writing this.

I don't know if they're right or not.


Thank you for reading this far, if you actually bothered. I know how big the tl;dr group is, and I don't expect most of them to make it here.

I expect by May, I will make a decision about it.

[Chance4U]

Well, I've never made a request before, so I guess this is from an outsider's standpoint...

That's an incredible weight for a person so dedicated to a fandom to bear. A lot of the people appreciate what ya'll do, and I want to thank you for taking on a burden like this.

To get it where the im@s compendium is at the moment took somewhere in the vicinity of about 70 hours for the base and for tweaks and updates, and at least 15 hours per catalog.

That's a ridiculous amount of work to do with only a few volunteers. In my opinion, you need to form a set team of reliable people (however many you deem is needed) and set jobs and goals for each member. After all, unity is strength♪. If others aren't willing to stand up and help you, then you shouldn't attempt this project. If people get burned out, it should be up to them to find a replacement.

I'd argue that for at least im@s: One for All that at least minimum wage should be paid for labour costs, which in turn will get moved to the im@s: One for All fund to pay for catalog purchases.

I did the calculations, and assuming we take a paltry 5 minutes per video requested (It takes a lot longer than that, but we'll assume that the other costs are magically free), it amounts to roughly 2USD per video. (Minimum wage here is about 16 dollars an hour, 12 videos, then dock paypal fees.)

Considering how much effort, time, and money is put into this, the fees need to happen. iM@S is expensive. If people want it available, then they really should help out with the cost.

I have no experience with a lot of this, but from what you've said, you need catalogers and programmers / coders. In an ideal situation, how many do you need? Could you hold a... sign up or something for this? And until you have enough people, the Compendium won't begin...? Just an idea...

Sorry if I'm being difficult... I'll go bury myself in a hole now...

[JNiles]

I'm just going to respond to random bits and pieces.


There's no shame in having an incomplete catalog, or a catalog that is built up in fits and starts.  Namco Bandai's ambitions aside, I don't think a single person should be held responsible for assembling a complete set of DLC.

Three years to complete a catalog for All for One would

You didn't mention anything about the hosting issues, but I imagine after what happened with Youtube, it's gotten worse.  I haven't gone link-hopping for Imas videos in a long time, but I figure Sony Columbia has laid waste to them by now.

Speaking of Youtube, I assume you have usage statistics for videos like you do for requests.  Do you find yourself creating videos that only a single person watches?  Is the effort justified?

Have you considered trying a streaming site like Twitch?  The quality would take a huge hit, the site is shaky at best, and I'm a little confused as to whether Twitch content is saved or vanishes into the ether, but gathering people with similar interests together is more efficient than serving up content one person at a time.  Basically, I'm proposing moving from an individual request system to a take-it-or-leave-it model where content is uploaded based on what you feel like doing instead of catering to a fractured base.  I've been getting into streaming and Let's Plays as of late, and a common factor is that people don't force themselves to follow a strict schedule or stick to the same thing all the time, so they don't burn out.

I'm speaking of course as someone who doesn't look for specific content in Imas.  I have no eye or ear for aesthetics so I watch whatever I stumble across.  So I am sometimes confused by the specificity of requests.

[Setsuna]

Well, I've never made a request before, so I guess this is from an outsider's standpoint...

That's an incredible weight for a person so dedicated to a fandom to bear. A lot of the people appreciate what ya'll do, and I want to thank you for taking on a burden like this.

That's a ridiculous amount of work to do with only a few volunteers. In my opinion, you need to form a set team of reliable people (however many you deem is needed) and set jobs and goals for each member. After all, unity is strength♪. If others aren't willing to stand up and help you, then you shouldn't attempt this project. If people get burned out, it should be up to them to find a replacement.

Considering how much effort, time, and money is put into this, the fees need to happen. iM@S is expensive. If people want it available, then they really should help out with the cost.

I have no experience with a lot of this, but from what you've said, you need catalogers and programmers / coders. In an ideal situation, how many do you need? Could you hold a... sign up or something for this? And until you have enough people, the Compendium won't begin...? Just an idea...

Sorry if I'm being difficult... I'll go bury myself in a hole now...

I'll walk through the im@s 2 experience we had, and the processes that are required to do it.

Well, the resources were there for im@s 2 - I got REALLY lucky the last time around, when Crisu stepped up. I haven't contacted him in regards with the One for All extension, because I don't want to pester him for something I may not commit to yet.

Due to the fact we CAN treat it as an extension makes it easier for him - a lot of the code is reused so his time is greatly reduced if we were to do One for All (And im@s 3, but we're getting ahead of ourselves here). Knowing what I know about the game though, there's some new things that need to be coded in logic wise though, but that's an odd extra hour or two.

The compendium logic is not that difficult (Except when starting up) because once things settle down, it's actually fairly easy to determine codes. The hard part is when a game first releases, because the order REALLY matters for the beginning since almost all request fillers use it (Namely the base game with no additional outlays) so there's always a massive scramble for unlocking everything.

Logic wise, there's usually an hour or two allocated towards figuring out the best way to carry over certain facets of the game. Stuff like how to carry over the order, (that's always a headache, because of how relatively unnatural it feels for the leader of a group to NOT be named first, but how weird it is to not obey the im@s 2 interface itself!) and how to read allocations of costumes and the like.

Granted, I DO like playing the im@s games, so that's more finding the actual hours to massively ace the game.

Most of the labour is in the manual collation of all the images, cropping them to size, then allocating them their codes, checking their previews, checking the allocation of the code works, running base checks and checking that there aren't any edge cases where the codes go wrong.

There aren't any edge cases in the THEORY of the compendium. The fun part is when you put it to software, there's always ways to break the compendium. For example, people thinking Xbox 360 accessories exist in the PS3 version, or how some songs can be put in quintet in the PS3 but not Xbox 360 versions of the games. We got this a LOT until we figured out how to separate them code wise and have it so that people wouldn't NOTICE that we had.

The usability of the compendium went up IMMENSELY when Crisu put in the javascript monkey hours to display and generate the codes for a user. Previously (Back in 2012) people had to generate those codes manually, and although straightfoward if you understood how archiving works, it wasn't simple. Codes couldn't be automatically played back to the request filler until 2013.

Fortunately, we're now in a position where we're getting close to where there don't need to be any more interface updates.

... So how much manual labour was required to populate all the images?

If you're wondering there's 39 songs, 140 costumes and 242 accessories in im@s 2, over 2 versions. Some are exclusive to that console.

Now, the fun part? There's 13 idols. So you can multiply all the wearing by 13, which brings you to a grand total of 4966 pictures to be cropped. (And 421 preview pictures.)

The compendium as of currently only has done about 1/5th of that. (11 of the 13 idols do not have their accessory preview picture. You'd think this wouldn't matter, but you'd be quite surprised how the idols individually wear their accessories quite differently, even though it's the exact same accessory!)

The guests have their own section, (10 songs, 2 costumes for each of Jupiter's cast, 2 miku, and 100 for 961) which adds another 104.

So all in all, you're only looking at about 5100 pictures. Grab, check the screenshot shows what it needs to, crop, then rename for the code to fetch, then recompress to size.

It's pretty time intensive to even do the grab phase to pass onto someone else to crop, mostly due to consistency checks, and the fact people have complained about the quality of the images.

In short? You'll probably get sick of cropping after the first 50 or so. I did, and I'm a very, very lousy cropper.

If you read through all of that and went 'Oh, that's not so hard', you'd be right. It's not hard at all. It's just very, VERY tedious and that's where the vast majority of the burnout comes from. I've been there. All the other volunteers have been there.

That's the labour costs of MAKING the compendium. This is before me (or anyone else) fills in any requests using it.

[Setsuna]

I'm just going to respond to random bits and pieces.


There's no shame in having an incomplete catalog, or a catalog that is built up in fits and starts.  Namco Bandai's ambitions aside, I don't think a single person should be held responsible for assembling a complete set of DLC.

Three years to complete a catalog for All for One would

You didn't mention anything about the hosting issues, but I imagine after what happened with Youtube, it's gotten worse.  I haven't gone link-hopping for Imas videos in a long time, but I figure Sony Columbia has laid waste to them by now.

Speaking of Youtube, I assume you have usage statistics for videos like you do for requests.  Do you find yourself creating videos that only a single person watches?  Is the effort justified?

Have you considered trying a streaming site like Twitch?  The quality would take a huge hit, the site is shaky at best, and I'm a little confused as to whether Twitch content is saved or vanishes into the ether, but gathering people with similar interests together is more efficient than serving up content one person at a time.  Basically, I'm proposing moving from an individual request system to a take-it-or-leave-it model where content is uploaded based on what you feel like doing instead of catering to a fractured base.  I've been getting into streaming and Let's Plays as of late, and a common factor is that people don't force themselves to follow a strict schedule or stick to the same thing all the time, so they don't burn out.

I'm speaking of course as someone who doesn't look for specific content in Imas.  I have no eye or ear for aesthetics so I watch whatever I stumble across.  So I am sometimes confused by the specificity of requests.

For compendium building reasons, you get weird things happening if you start building them peacemeal, and things can get complicated because of it.

You can normally guess the number a particular item is, if you know what the others in the set was. NBGI have been (fairly) good in im@s 2 to bundle accessories. In im@s L4U, the numbers get stupidly variant, to say the least.

It gets confusing (even if we move to a 90%+ automatic system, there'll have to be people to maintain the integrity of the code and allocate new ones) when you're doing checks on the code, and say Hd-29 is the Vampire Girl head accessory, but Lg-41 is the leg accessory of the same costume. You don't want to be in a position where you'll be constantly scanning code, hoping that you didn't forget an accessory.

Doesn't sound like a big thing, but when you're trying to ensure that when people are talking about something in the same language (which what the compendium actually IS - an ability for its users to go 'I want this' and the fillers to go 'You want this' and they always match) you want to make sure its as consistent as you can make it.

If nothing else, if I DO reshuffle them back into their spots later, people will go 'Wait, why did the code numbers shift?' and then we get confused people (who usually complain how they didn't get what they want)

-----

For Youtube statistics, you normally see between 300-500 people watching a video over its lifetime. At peak, I had 1000 per video, and I DO have one particular video that cleared 35000 people accounted for (Apparently it went to 4chan).

Since I don't own NBGI's material, NBGI probably was paid 4 dollars for the ads (ads are paid at a 1 dollar per thousand, and usually 20% of all watches have an ad delivered to them due to adblock/selection etc.) for that video, and has probably made a couple of hundred dollars over the history of the service. It's chump change to a corporation that big, of course.

Yes, request fillers don't make money from any ads that display. It's probably better than banning them outright, so it's part of the deal.

Surprisingly, the hosting part isn't really a problem. Sony hasn't actually been interfering all that much with it - A few recent DMCA counter-notices I issued forced Youtube to acknowledge that the videos are NBGI's primary property and were produced USING that property, not Sony's, and subsequently, fall under NBGI's content ID umbrella instead. It's not reliable, of course, but I have had words with Youtube's legal team before.

You'd be surprised how easy it is to get the files around regardless.

The real problem of using twitch.tv and other streaming services to provide stuff like this is to do with the quality of the playback.

I'll try and give the cliff notes version, only because most people don't want to know the technical side:

- A user's upload is generally a fraction of their download. Basically odds are you can download about 10x faster than you can upload barring certain circumstances. Depending where you are in the world, this number can go up to as high as 35x.

So as an example, I have a connection of 12mbps down/1.4mbps up. In Australia, it's considered absurdly good for the type of service I'm on, which is the best service Telstra will provision at the moment. If I want to improve it further, I will have to pay about 4x what I do now to increase those numbers by x2. What do I pay right now? 100 AU a MONTH.

(If you want me to move to fibre, the cheapest way for me to do that is to spend 400k and buy a house. No, I'm not kidding. Telstra WILL flog you with the installation costs end to end if you even SUGGEST you will pay for fibre installation, and since that includes backend equipment, that's only a couple of mill.)

Given an 'on demand' stream like twitch, you'll be able to encode and send at (say) 1.2 mbps up.

What's your average youtube video encoded at? Between 3-5 mbps.

So essentially, given absolutely nothing else, someone doing it properly (sending video far better to youtube so that the loss on reencoding is minimal) can take all the time in the world, and ensure that youtube video (or any other video service) will look as good as the service it's hosted on will allow it.

There's a whole bunch of other factors too (namely, how long it gets to encode, how it handles gradients and other things)  but usually on a good day even if the video encode was at the exact same (Namely, we went 1.2 mbps for the video and compared it to the stream) it'd probably look significantly worse.

There's a few scenarios I can name which will make that twitch stream straight up unwatchable.

There's also other considerations, usually to do with if you want to use the video to make another video, even though I keep telling people to NOT use youtube video to make their videos with.

But what can you do? They're starting out, so they'll get better... eventually.

They're usually the times when precision is key though. Your average requester isn't always that picky, although from the stories I've heard, sometimes I'm not convinced.

-----

As for the One for All first catalog joke, I just did the calculations.

If I assumed that I would be donated 30 dollars over 2 years, and that the game will discount DLC 2 years after release, the math goes that we'd be able to get about 50 dollars (about 7000 yen) to pay for the discounted catalog 1 in 3 years, assuming their history holds in regards with DLC.

And there's one ruined joke...

[Byuusan]

As a person who uses the compendium for pretty much everything on my channel, i have ot thank you and the crew for the work done on this.

I never really learn how much work it was to make it, as i was pretty late into it, and all i really did to help was confirm the PS3 in-game accessory and song arrangement.

As much as i wanna help with the development for One for All, ultimately, i can't really do anything.

Funding on my end is really tight now that i own a house and need to pay it off, as well as other bills and stuff that comes along with that.

although i can attempt to help with anything that not related to money, i can imagine i might be slow between RL stuff, and requests.


Ultimately what i planned in the end in the case it never get made into a reality, make a personal Compendium on my..... blog.......... yea that not going to get done till at least 2015.

On the side note about the Youtube thing.

For MOST songs i have no issues with sony. Barring 4 songs (READY!!, Nana Iro Button, Ai like Hamburger, Kyun! Vampire Girl) and ad are setup on them, course with NBGI taking it all. Course way better than being straight up banned.

my video generally get anywhere between 150-500 views in a lifetime. From time to time, i get videos reaching 1k to 5k views (where they're going i have no idea.). Only one video to date that has 124,918 views as to this post release. That being said that was a special videos and outside the specs of the Compendium.

[Setsuna]

After a few more discussions with a few people I know, I am going to implement a system in an attempt (as much as I and several other people don't expect this to work) to raise the money required to cover the catalogs for One for All, and to cover development time for the compendium, time costs for various people, server time etc.

New rules going forward (18/03/2014)

- All requests will be filled at a cost of 2USD per request. This is for all types (L4U, im@s 2 PS3 and im@s2 X360).
- No requests will be entertained otherwise. You can try your luck, and I am free to decline, just like anyone else.

For multiple requests, discount rates will apply:
10% for 5 requests or more.
25% for 10 or more.
33% for 15 or more.

A membership style of deal may be considered, subject to various limits. That is still being considered.

This is a charge for the TIME taken to process requests, not of the items. (This is in fact legal, and can be billed.)

For any bulk purchases, there will be an acknowledgements page, as well as order priority for costumes and the like for One for All.

All requests covered in this manner will be undertaken in 48 hours of acknowledgement of receipt of payment.  Turnover time is subject to a maximum of 5 per day, so if you ask for 40 for some unknown reason, don't expect them all done for at least a week.

All money raised will be put towards costs of catalog purchase, time compensation, and other costs. All money will be accounted for publicly (probably in this thread, or a new one).

The costs will be reduced when it is believed that the costs of the compendium have been fully covered. You probably don't want to know how much the estimate is though, considering I'm looking at the price sheet for Catalog 1 for One for All right now.

You are free to use the compendium to request from other parties. It is their decision to fill a request.

Please PM me or email at night@kekkai.org for payment details.

Thank you for your attention.

[DeviantProtagonist]

Bold decision. While I share said doubt in this way of financing, if it means people can have a hand in giving along with taking then I'm all ears.

[Byuusan]

Im also doubting this is the best way to finance this, but this is however in my mind, the best way as well.

[Chance4U]

Well, with the announcement that you can officially have multiple costumes per song...

I just don't think this is feasible...

[DeviantProtagonist]

Never say never. For all intents and purposes, there's instead the phrase, we'll make it work.

Not only that, I imagine the existing compendium makes for a dandy steppingstone.

[Setsuna]

Well, with the announcement that you can officially have multiple costumes per song...

I just don't think this is feasible...

Oh, I've known about that for months. If and when the project goes ahead, it has already been accounted for.

[Byuusan]

It been confirmed that multiple outfit setup is possible in one song/video.

If we're going to implment that (we should though), how we going to do so?

[Setsuna]

It been confirmed that multiple outfit setup is possible in one song/video.

If we're going to implment that (we should though), how we going to do so?

I'll need to talk with you in the coming weeks actually - I'll be arranging for a prototype to be built in the next few weeks which will address that problem, and a few others.

I've already got a paper design working, but that's actually going to be small fry compared to the rest of it...

[AnimeMagic]

Hi Setsuna. I don't want to sound rude, but I just wanted to ask you If you where aware that charging for requests is illegal because it's technically selling copyrighted material that you don't own and you could be arrested and/or sued by a number of people including Namco Bandai, Composers, etc,?