non-IOE by zhaoce (米高蜥蜴)

发信人: zhaoce (米高蜥蜴), 信区: Military

标  题: 天朝现在风起云涌的去ioe运动

发信站: BBS 未名空间站 (Wed May 28 05:41:48 2014, 美东)

谈谈去 IOE 运动

这篇文章算是今年年末的一个技术总结。谈谈技术圈一度的热门话题「去 IOE」这件事。

何谓 IOE ?

所谓 IOE 是个简称。是指以 IBM 、Oracle、EMC 为代表的小型机、集中式数据库和高

端存储的技术架构。其中 I 指 IBM p 系列小型机,操作系统是 AIX,IBM 专有的

Unix 系统;O 指 Oracle 数据库(RDBMS);E 指 EMC 中高端 SAN 存储,曾经一度是

IT 企业很喜欢采用的技术架构。IOE 这个说法怎么来的? 据我所知应是来自阿里技术

团队内部的称谓,然后才在整个业界流传开来。如果你去问国外技术专家什么是 IOE,

对方肯定一头雾水。当然,随着国内案例逐渐被介绍到国外,或许某一天这个术语能输

出价值观也说不定。

在小型机领域,只有 IBM 这一家,独步武林;HP 当初把宝押在安腾上,算是早早退出

这个市场;Sun 日薄西山,SPARC 机器…那就更不必说了。另外,需要说明的是,IBM

也生产存储产品,但 IBM 的存储产品早期其实挺山寨,竞争不过 EMC ,而且有些用户

会忌讳把所有的东西困在一家公司身上,尾大不掉。 起码在国内,EMC 的占有率应该

更高。中高端存储这个领域,还有一家 HDS,不过曾经一度在阿里也栽过跟头。数据库

软件方面,在当初几乎没的选择,只有 Oracle 这一家,IBM 的 DB2 实在是不行,虽

然号称市场占有率不错。国内用 Oracle 数据库支撑互联网应用的话,一般是采用

Data Guard 这个架构方案。

为何要「去 IOE」?

说起「去 IOE」,跟阿里的王坚博士有直接关系。我无从得知他当时为什么要做出这个

决定。但根据我的推断,当时淘宝、支付宝等公司每家技术体系各有特色,技术团队也

各是一套,只有去「去 IOE」,才有可能将淘宝、支付宝等公司的网站核心体系架构迁

移到云上,体现阿里云的价值,某些管理者才有可能从集团公司层面对整个技术团队有

更好的控制力。否则,阿里云师出无名。注意,这个说法只是我个人臆测,肯定不是事

实,只是逻辑上是说得通的。实际上,阿里云当时自己的活儿做的很垃圾,也幸亏这个

「去 IOE」运动进行并不那么快。当然这是后话了。

或许有人认为「去 IOE」会节约企业成本,实际上,当时的 Oracle 和 EMC 等软件成

本已经足够低,硬件上,硬件上的每年的成本也是可控的,如果考虑迁移后总体成本,

新硬件成本、开发人员成本、运维成本、时间成本等等,通通算下来,未必能节约多少

。这个不是我拍脑袋给出来的,而是跟不少技术人事后复盘,结论基本一致。

客观的说,当时「去 IOE」有一种公司政治的倾向,或者成为一个一窝蜂的运动,这很

令人讨厌,或者说这事情出发点未必如何好,但令人意外的是,最后在阿里诸多优秀技

术人才的努力下,却取得了一个令人惊讶的很好的结果,那么,就别管出发点如何了。

为何「去 IOE」是必要的?

从另外一个角度考虑,尤其从运维 DBA 的角度去审视,「去 IOE」 实际上是必须要进

行的,或者说去「O」是必须的,因为当时存在的问题是,Oracle 数据库对用户 (DBA)

来说已经不够灵活,常用的 Data Guard 模式无法适应互联网公司快速增长,最基本

的一点,读写分离就做不到,只能向上扩展(Scale Up),拼硬件能力,几乎无法做到横

向扩展。或许有人说,不是有 RAC 么? 但 Oracle RAC 是无法对付高并发下的 OLTP

应用的 – 一直到现在很多人都认识不到这一点,RAC 跑跑数据仓库什么的倒是不错。

注:有人会说 Orale RDBMS 11g 的 Data Guard 可以读写分离呀,这个所谓的读写分

离可靠性其实是不够的,而且出现的时间也太晚了,此外,不够灵活。还会有人争论

Oracle RAC 怎么就不能应付 OLTP 呢? 别争论了,你非要说可以应付,没问题,但是

在阿里体系的公司里,还真没人敢这么玩儿,为什么? 是做不到? 还是他们掉进坑过?

如果要动「O」,那么 「I」 和「E」就必须要动 – 相信不会有人在小型机上跑

MySQL 的,而且,只换掉「O」也没有意义,换汤不换药不会有成效。

随着中国电子商务的快速发展,整个阿里系其实已经在面对全世界增长最快最复杂的业

务系统之一,这是机遇,也是挑战。旧有的技术架构已经不足以支撑更大的梦想。从这

个意义上来说,去「IOE」是相当必要的。或许,这也是王坚博士以及一些人的初衷。

为何「去 IOE」成功了?

阿里几家子公司这么复杂的技术体系,「去 IOE」这事情堪比高速公路上给飞驰的汽车

换轮胎,最后成功是相当不容易的。

成功的因素有哪些呢?

1.功不可没的当然是一群出色的技术人才,很了不起。我想这是无需多说的,面对这么

复杂的业务环境,这个任务如果没有一批优秀的工程师是绝对做不到的,没有阿里 B2B

技术团队、淘宝团队、支付宝技术团队的先后投入以及合作实践也是绝对做不到的。

要强调一下的是,阿里在在分布式事务上的处理能力和解决方案,这应该是独门绝技。

在业界各种会议上也经常能看到这一群牛人出来分享,同行应该能感受到。

2.开源软件的快速成熟。举个例子,这两年 MySQL 体系的软件进步相当惊人,各种经

验证的解决方案如雨后春笋般涌现出来。这得益于不少知名互联网公司(比如 Facebook

、淘宝)在使用 MySQL 的同时也将其技术改进回馈给技术社区,把技术方案分享给业界

,业界在吸收这些技术的同时再次回馈给技术社区,形成正向的反馈,极大地提升了开

源软件在商业领域的竞争力。

3.硬件革命。硬件的进步给技术体系的变迁做好了铺垫。最主要的关键词:「SSD」。如

果没有「SSD」的技术成熟以及在商业应用上被普遍接受,「去 IOE」几乎是不可能做

到的。要知道物理机械硬盘存储的性能数十年几乎没得到什么大的改进 – 当然每年提

升一点是有的。但 SSD 相比机械硬盘来说,则是质的飞跃。我记忆深刻的是,每年做

I/O 容量规划的时候都会发愁,因为即使已经使用上了很高端的 EMC 存储设备,但实

际上只要应用层 I/O 没有命中到存储内存,直接打到后面的磁盘上,几乎没什么抵抗

能力。比如当时一个硬盘极限能撑 100 多个 I/O,100 块硬盘也不过是万把个 I/O 就

不行了。 但这样的 I/O 「打击」对 SSD 来说,则不是什么大问题。SSD 给解决「IOE

」体系最大的瓶颈 – I/O 能力提供了硬件先决条件。

4.摩尔定律。这一点最初我不想提及,但不提及,就会有别人说,所以还是补充一下。

提到摩尔定律,重点要说的 X86 芯片的计算能力不断进步,而 IBM 的 Power 芯片虽

然也在不断进步,但正式商用的节奏明显在控制。这就给 Intel 体系带来了机会和空

间。

国内对「去 IOE」的反应

在出现阿里这个成功案例之后,技术圈很是震动,曾经一度讨论热烈,随后则是国内产

业界对此出现了一些跟风的倾向,不少公司则打着「国产」软件的旗号出来蒙人,这是

值得警惕的。去掉 Oracle 不意味着就要采用国产的垃圾数据库,因为 MySQL 以及衍

生的各种分支数据库才是最佳选择。同样,不用 IBM 的小型机也不意味着国产服务器

就迎来新机会,在用户那里,适合的解决方案才是最重要的。「去 IOE」不应该成为一

个噱头。任何时候,「国产」都不应该是一个互联网企业选型所要优先考虑的因素。在

全球化的今天,我们应该忘掉「国产」,才有可能早点做出来更牛的软件来。

更好笑的,还搞出来一个什么「去 SOA」的组织,我觉得这是不太正常的,实际需求为

前提,不能本末倒置,难道是为了「去」而「去」么?

2014 以后会有更多公司「去 IOE」

从目前的种种趋势来看,在今后几年,国内一些互联网公司以及 IT 企业会逐渐的「去

IOE」化。相比几年前,现在的「去 IOE」的主要原因则是:旧的「三件套」已经的确

不适合互联网应用了。开源数据库更为可靠成熟,SSD 可靠性也得到验证,技术人才甚

至都不需要从头开始进行储备 – 类似「沃趣科技」这样的团队已经能够提供足够好的

技术支持服务,新的技术体系毫无疑问会让企业更有竞争力,总体成本更低。

上文提到的「沃趣科技」是由一群前阿里的工程师组成的技术团队,汇集了一群从数据

库到存储到网络架构的专家,如果要找「去 IOE」技术顾问,似乎他们是独一份(这里

不是广告)。相比之下,IBM、Oracle、EMC 等公司近些年来,实际上对国内那些快速发

展的互联网公司已经提供不了有力的技术支持了,IBM 拿苏宁电商练手更成为业内笑柄

。或许这也是 IOE 们被抛弃的一个原因,也可能是一些创业团队的新机会。

一个时代过去了。

–EOF–

关于 IOPS 的数据补充:

机械硬盘现在最高号称能跑到 400 IOPS,但应该 200 左右(走 SAS 接口)也就是极限

了; 单块 SSD(走 PCIe 接口)的最高记录是九百多万,用不了多久突破千万 IOPS 是没

问题的,相当了不起, 即使百万量级也足够吓人了。(Refer)

Advertisements

Tips for load test by using Visual Studio 2012 Ultimate

Tips for load test by using Visual Studio 2012 Ultimate

 

Have some time to use the visual studio 2012 ultimate edition for load test recently.  So there is some basic tips about this.

1.      Only the ultimate edition VS 2012 has the web performance and load test ability.
Trial version VS 2012 only support Max 25 user load.
You must install all the VS update for the load test. As first version VS 2012 has many bugs in testing. Current latest UP is 3.

2.      You can follow the walk through of the Microsoft to step by step create the web test and load test.  This is fast and best way to start learn recording and concepts.

http://msdn.microsoft.com/en-us/library/vstudio/ms182551.aspx

http://msdn.microsoft.com/en-us/library/vstudio/ms182552.aspx

http://msdn.microsoft.com/en-us/library/vstudio/ms182594.aspx

And I have some tips can make you understand advanced load test skills quickly:

 

How to deploy test used resources files for the file test?

Such as you want test upload file feature , so you need source file to use,  when the web test is recorded, web test will not use the source folder of yours anymore. So you have to put the source file into the deploy folder of your test project. What you need to do is to add the folder or files into your project, by add files from “local.testsettings”—deploy, you can realize this.

http://msdn.microsoft.com/en-us/library/vstudio/ms182475.aspx

 

How to make the web site host URL as the parameter input for web test?

There is feature called “Parameterize Web Servers” feature in the web test, can auto extract the site host name and port number from the requests. This can save the time if you want switch your test from a UAT server to a production server.   What you to do is just modify the “WebServer1” and other  parameters to change the host name.

How switch test from a http site to a https site?

If your target has enable the https, but all site content are same. What you need to do is change the “Parameterize Web Servers” data to make it as https, you do not need to re-record all your request.

What you need to do from a web test to a load test?

1.      Your web test may run fine if you just run once every time, but start to run it as the load test may get issues as concurrent user will cause the conflicts if user conflict at the features, such as check in and check out same file, rename/delete same file etc.

2.      A load test recorded by default will create random unique resource names, such as you upload a file, load test will create unique file name for you at the upload request. Open your request and check the  file name , under the properties, there is some thing called “Generate Unique Name” to control whether or not create unique name for each request.

3.      After you have the web test, and you prepare the data source csv file already, then add/create DS into your test project, you can use the DS data in your load test to let test use the file name from the DS. By this format{{}}, what you need to do is to go through each request data to check if any thing can be changed/inserted by the DS data, by this way you can create a mean full load test. Such as each user load create separate file/ rename/delete/check in /check out separate file. File named is dynamic defined from DS csv file: this will avoid load test failed by conflicts.

4.      In load test, there is some thing important, one is the “load pattern”, you need to understand “constant load ” and “step load” pattern, it will be very useful. And under the “Running Setting” , the “Run Duration” and “Test Iteration” is very important.  Iteration here mean run once the web test, when a load test start to run, each user load will run the iteration again and again, unless  the Max Test iteration reached or time duration is reached. This is basic idea of the load test works.

 

Data drive test by data source – how to create a csv file and use it as data source for your test request?

You can create a Data source into your test project by a CSV file:

username, password
user1, password1
user2, password2
user3, password3
user4, password4
user5, password5

then you can use the data by binding call, {{ UserDS.UserNameFile#csv.username}} in the request data input. Check each request data and web server XML input to make sure the data is changed upon your needs.

http://msdn.microsoft.com/en-us/library/ms404707.aspx

http://msdn.microsoft.com/en-us/library/bb385971.aspx

Data source access methods explaination

http://blogs.msdn.com/b/slumley/archive/2008/03/14/description-of-access-methods-in-data-sources.aspx

How to create different user account from a data source (csv file) to log in to the page?

You can create a csv file with user name and password inside, and use it in the web test as the Data source, to load test will auto log in with these users during the test, please note this does not means the “user load” in the test. Even you only has one user account , user load still can simulate hundreds concurrent user load for you in test.

http://social.msdn.microsoft.com/Forums/en-US/80c15dfd-5265-4e9f-becc-5e13fdf19596/how-do-i-configure-multiple-users-in-a-load-test

How to create a plugin to do each user use each row from DS?

http://social.msdn.microsoft.com/Forums/fr-FR/5853028c-d297-45fb-ba46-ae346fa1e21c/load-test-datasource-distribution

how to create a plug in

http://msdn.microsoft.com/en-us/library/ms243191%28v=vs.100%29.aspx

Difference between the concurrent users load and VS multiple account?

http://social.msdn.microsoft.com/Forums/en-US/5829e774-f552-42e9-b43d-46daaaa4e6a0/load-testing-for-concurrent-users

how to debug a web test

http://blogs.msdn.com/b/slumley/archive/2007/04/10/how-to-debug-a-web-test.aspx

 

How to install test controller and agent

If one test client is not enough, you can install the controller and agent to make several client PC to make the test.

http://msdn.microsoft.com/en-us/library/vstudio/hh546459.aspx

http://msdn.microsoft.com/library/hh546460%28v=vs.110%29.aspx

 

 

Oracle Distributed Document Capture 10gR3 install and basic setup

Oracle Distributed Document Capture 10gR3 install and basic run setup

By W.ZH Apr. 2013In Oracle Distributed Document Capture, users scan groups of pages in batches, using

settings saved by an administrator in a scan profile. A batch is divided into one or more

documents, and each document contains one or more pages

There is few doc article online give how to install the ODDC and how to start use it. This is just so poor product cared by somebody.

(ODC at this time is not available download from oracle website. maybe they are doing dev for a real 11g version.)


=============== How to install ODDC =========================================================

1. download the installer of ODDC

2. install ODDC on windows, before that you need to install the JAVA for it. please note that a 32bit java is need. then set the PATH for java bin folder in windows.

unzip and just run the setup to install it.

3. you need to read the install guide and do the basic configuration:

After install, what you can get is the several things:

    1. A program called “Oracle Document Capture”

2. A program called “Distributed Document Capture Server Configuration”

    3. A service in windows, registered as “Oracle Distributed Document Capture”
 

Here are things you need to config to make ODDC can be ready to work.

1. Go to “Oracle Document Capture”, to setup the  Batch folder / Commit folder / Capture Database config

for those folders, you can create a main folder to contain them as there are still more folders needed later.

for the Database config, refer to install guide, simple way is to use the access DB first.

if you can not find the “Capture.mdb” under the <installed Dir>, then you can go to C;\ProgramData\Oracle\Document Capture\, copy it out and save it to <installed Dir>, then use this version.(do not point to that runtime version!)

2. Go to  “Distributed Document Capture Server Configuration”,

set those folders, like,  Error Batch location  and  Log file location.

The “Set Client Access Folders” is the folder named “ClientAccess” under the <install dir>\WebPages\   , this folder is going to be the location that client upload their PAK files to web server.

The others just follow the install the guide.

    3. Configuring the IIS server,

You need to start a IIS server in your windows.

Enable the IIS manage console, and also setup the ASP and related filters, just follow the install guide to do these.

Then create the application pool and a website(or a virtual dir) in the IIS,

Deploy the <install dir>\WebPages  folder to your new created website.  we suppose the website can be     http://10.10.10.10:888/***  as there is no index.html in the folder, just point to any file to make sure your site runs.

    4. Give folder permission,
there is a user “IUSR” in the windows, work for IIS, you need give all the folders that we have setup in step 1,2,3, include <install dir> the read write permissions
    5. let us make a verfiy,
Try to access admin URL of ODDC.
http://10.10.10.10:888/WebCaptureAdmin.html

            If you can login and see the admin UI (by Java applet?), then your IIS works.   we call this UI as the “Scan Profile Admin UI” later.

To this step, you seems set all things done, but still not able to understand that how system work…. what a user unfriendly product!

=============== How to make the ODDC run with your UCM 11g ===================================================

1. Make your Webcenter Content (UCM) installed and runs.  After that a basic ODDC example will need you to config “File Cabinent” + “Scan Profile” + “Index/Pick List (Optional)” , then Client can access to server side to run.

2. Go to “Oracle Document Capture”,– admin to config the “File Cabinent”,
    under “Commit Profile”, you an set it to link to UCM server or just save the received file to a folder.
    for UCM, you can also create index fields and map them to the UCM internal metadata at there settings.

    you can also setup the pick-list or DB lookup for index at here. if you have index asked, then later when client try to scan/send file, it is asked in input/select the indexed data together with the doc.

3. After you have the “File Cabinent”, then you go to
“Scan Profile Admin UI”, to use this File Cabinent to create a scan profile. just follow the admin guide to do these.

4. Go the windows services, start the service of  “Oracle Distributed Document Capture”

5. finally you can go to Client to do a real scan and send/upload.
        find a 32bit IE browser, as the web page need to run Active X control. Open the client URL like this: http://10.10.10.10:888/WebCapture.asp
after login you can follow the user guide, select the scan profile and to do the scan, input index data and also the send, etc.
5. check out the batch folder, ClientAccess folder at the server side. File will upload to IIS as PAK file under the ClientAccess folder, then the capture service will unzip it and process it in the batch folder.
after that, as we have config the commit to UCM. you can go to UCM to search the new checked in files.
and in the UCM, check the metadata of your files, if you have set the field mapping, then your index data will be checked in with your file to the UCM too.

        And also read the log in the log folder.

Issue and solution:

1. when I set up the UCM 11g as the commit server, i get error of creating the Capture-UCM Implementation Object”

Solution: your windows do not have java installed, install the java and also the set the PATH.refer to https://forums.oracle.com/forums/thread.jspa?threadID=1091622

2. After i setup every thing, i can go to client to import and send file, and it shows successful, but i can not find the file checked in the UCM side.
Solution: you should check the folder of the ClientAccess first, if you find the PAK file inside here, but the batch folder has nothing happen.

then reason should be you do not start the service in the windows. just do it and can fix this issue.

Please note that this service is not mentioned in the admin and user guide , and in install guide not clear too…. What a mess of oracle docs!

Reference:

10gR3 ODDC install/admin and user guide

        http://www.youtube.com/watch?v=4PwfGLw44oo   a very good video – how to config the ODC (not ODDC)   file storage etc.

https://www.youtube.com/watch?v=uYC-O0t30gc    some help about the access db.

Notes:

Only ODC support the The PDF (Searchable) option uses Optical Character Recognition (OCR) when creating PDF documents that users can search.

Note that the hidden text (the OCR results) within PDF (Searchable) documents CANNOT be edited and may contain errors.

Instead of access DB, you want to use the oracle DB, then you have to install the OLE driver for oracle, which is not in the software by default…I do not know why they do not like oracle DB.

Scheduled Process Management in Fusion Applications

By W.ZH Feb 2012

 

Issue:

In my last article ESS-01050 error when you try to create a new scheduled process, I have shown one issue you may face in the scheduled process management; we can simply say the reason for that issue: Scheduled processes are confined by user and your role; you cannot see other user’s process. But system level, when you create a process, will detect the duplication process cross the user context. This is the reason you see that error message.

After you use the Scheduled processes page for a while you may find it have some limitations here. You can do create new process/resubmit/Put On Hold/Cancel/Release Process/View Log

Currently I do not try out on the Resubmit/Release Process feature yet, but I suppose they are not an update feature. We need some basic knowledge about the ESS here (Oracle Oracle Enterprise Scheduler Service). This scheduled process UI in fact use the ESS as the backend to trigger the job by the timer scheduled. For example, a daily job run at 5AM, when it will run tomorrow, at today 8AM, you will see its status is “Wait”, when it starts run at the time of tomorrow 5AM, you will see the status is “running”, after it is finished it normally shows “Succeeded”.

When it is in “wait” status, you can use the button to cancel it, then it will change to “cancelled” status, but as it is a daily job, so system will auto create a new job with default status “wait”. This new job scheduled to run at the day after tomorrow 5AM. So you must be very careful at here. As there is no update feature, such as, for this example, if you want to run this job today immediately, how can you do this?

You cannot create a new process with same task, as system does not allow duplicate task. You cannot change this current exist scheduler to let it can run immediately as system UI has no such feature. In fact, you stuck at here.

Reason:

Because there is no change scheduler ability, at least I do not find it now.

Solution:

You can use the EM to solve this issue, this need the WLS admin user account. All the Scheduled processes are the schedule jobs in the ESS server. Log into your EM for your domain management, under the ESS, you will see how to manage the scheduled jobs, each job with ID that can be matched to FA UI page. For a daily job, you can only change the schedule by going to the root job to change the whole schedule. Each daily is just a sub-job under the root job.  I think I do not need to detail all the feature in the ESS admin ability, you need to refer the ESS user/dev guide for that.

How to save Media File(Copy)

How to save Streaming Media

From http://www.infocellar.com/software/streaming-media.htm

– for FREE or for a small fee –
(some files are impossible to get without a streaming recorder utility)

*** also see http://all-streaming-media.com/record-video-stream/

FLV Files (FLash Video)  –  FLV files are rather new but they have quickly become all the rage with these video clip sites where kids do dumb things so they can get hurt and show the world.  FLV’s were created for streaming – but they can be delivered in three different ways:

  1. Embedded in a SWF file using the Flash authoring tool (supported in Flash Player 6 and later)
  2. Streamed via RTMP to the Flash Player using the Flash Media Server (formerly called Flash Communication Server)
    This is the best option for allowing advanced seek capabilities and automatic bandwidth detection.
  3. Progressively downloaded via HTTP (supported in Flash Player 7 and later.)

Summary – How to grab the Various Types of streaming Video Files

*** FIRST *** always check to see if the site has provided a “Download” link, and if they have . . . right-click on the link and select “Save Target as . . .”

  • WMV Files – there is not other way . . . so just buy and use WMrecorder ($30)
  • Real Media – first try the manual method below – if that fails (and it usually does fail, especially these days, then use RMrecorder – however, RM Recorder cannot record Real™ streaming video and audio (RM, RA) streamed through the following protocols: HTTP RTSP  -***  instead of buying both WMrecorder and RMrecorder ($60 total) – you might as well buy the full version of WMrecorder for $50.  It records almost ALL Formats –  WMV, RM, Flash Video, Quicktime (MOV), Mpeg, MP3 and others. Also includes Scheduling option.
  • Quicktime (MOV) – use the “indirect” save trick (see below) – if that fails try MOV Recorder
  • Google Video (gvp files) – these are just tiny text files – free download is hard to explain but simple to do (see instructions below)
  • Flash Video Embedded – with an FLV (FLash Video) file embedded into the swf file – all you need is to download the swf file – no need to “capture the stream” – so just use Flash Extract Pilot (free but only available here)
  • Flash Video Streaming (the swf file links to an FLV file stored on a server) – these are the toughest types to get !!!  YouTube uses this type of video clips, as do a lot of other sites.  The flash file (swf file) is just a small file that presents a player window for the video – but the video is linked within the swf file and is stored on a server somewhere else.  Unfortunately the URL to that location is hidden and almost impossible to find on your own.  But “Replay A/V” will find the URL for you !!!ALL YOU NEED IS THE DEMO !!!  You can download the demo of Replay A/V, which comes with an FLV player.  Run it, and click “Stream Capture”, click “Clear List” if anything is there – and then go to the site with the video on it, and as soon as it starts playing click the Stop or Pause button (no need to play the video now), and you will see the URL’s listed in the window, as shown here:

    The Replay A/V demo limits you to 5 MB downloads.  However, it does list the URL of the FLV file for you !!!  So you really only need to use this utility as your “detective”.  In this first screen (above) the text cannot be selected and copied – so click once on the FLV entry to select it and then click “Add Selection as New Recording”.  The following window will pop up – you can then copy the URL and paste it into IE and then you can download it from there !!

    NOTE:  Replay A/v is actually a multi-stream recorder.  It can capture – Windows Media (HTTP, MMS, RTSP Real: HTTP, RTSP), QuickTime HTTP, Flash (SWF and Flash Video), FLV, AOL Video (NSV), Shoutcast/Live 365 (streaming MP3), and SlingBox™ streams.

    Converting FLV Files to MPG !!  I use “Total Video Converter” and convert them to MPEG-1 usually, since we are dealing with desktop videos (not high quality clips) – but there are several others that will work

    Converting other Video files to FLV (i.e. create FLV’s for your own website) – use Riva FLC Encoder

*** there is one utility called HiDownload that they claim can capture WMV, MOV, and RM files – however, no one I know has tried it !!!

Saving Streaming Media without any special Software

Every now and then you come to a page with a fantastic flash movie, or a Real Audio song that you love, or a Windows Media video clip that is exactly what you have been looking for.  No problem, you figure, “I’ll just right-click on it and save it like I do with images”.  Woops, doesn’t work with streaming clips !!

Fortunately, the majority of these clips, while they stream in . . . are being saved to a file on your hard drive.  This file is usually in the “Temporary Internet Files” folder. When the clip is finished loading, the file usually stays there until you move to another web page or close your browser.

       But I don’t see any video Files in my “Temporary Internet Files” Folder ??

In some cases the clip is either saved somewhere else (who knows where? – but check the “temp” folder) or is stored in memory (RAM), and therefore no file is created and you cannot save it !!   For example, CBS.com clips use an integrated Internet Explorer Real Player, which plays clips in RAM only.  But most clips are stored as a file, since RAM is so precious..

If a clip is not stored as a file, and you absolutely need it – film it with a video camcorder and if you have a video capture card, import it as an avi file.  The quality will be terrible and you will get the refresh screen flickering – but you will be able to view it later.  This is not recommended !!

Where is the “Temporary Internet Files” Folder ??

  • Win98-ME  –  “c:\windows”
  • Win2000-XP  – “c:\documents and settings\user_name\Local Settings\Temporary Internet Files”

IMPORTANT – with WinXP when you right-click on the “Temporary Internet Files” folder, there is no Search option in the popu menu.  So instead, right-click on the “Local Settings” folder, one level up, and there will be a Search option.  To search for all streaming and standard video files, in the “Find” field, enter:
*.avi,*.mpg,*,*.asf,*.wmv,*.wma,*.rm,*.ra,etc.

How to Save the Various Types of Streaming Media

see also  http://www.streamingmediaworld.com 

and for short (120 kB) ASF and RM sample clips – go to http://www.kidsmusiconline.com/sample.htm 

for Audio – simply Record it !!
If all else fails you can usually start up an audio recorder utility such as Cool Edit, and record while the streaming clip plays.  If you have a dialup connection and it stops to re-buffer, you can carefully edit out the silence in the clip.

The Linked Streaming Media Links “trick”

the “Create your own Web Page” Trick – for Quicktime (*.mov files) and Real “*.RAM files”

If a link on a web page references a streaming video file directly, you just right-click on the link and select “Save Target as . . .” and then save the file.  BUT quite often the link on the web page will play a video, but indirectly.  It may take you to a page with the video clip embedded (Quicktime), or it may have a pointer text file, such as Real Audio/Video RAM and SMI files.  Here is the trick:

  • right-click on the link and select “Save Target as . . .” and save the file.  Real files will be RAM, or SMI.  QuickTime files will usually be an HTML file.  If the original webpage has a direct link to the QuickTime file – no problem, just right-click that link and save it.
  • open the file in a text editor such as Notepad
  • find the URL location of the actual video clip – if it is a partial link (relative path) such as “video\clip1.mov” then you will have to try appending that to URL of the original site (if no worky, then try appending it to the root web URL, or one directory level up from the original site URL – finding a partial link is trial and error and is impossible many times).
  • now – at this point, even if you open your browser and enter the location – the clip plays but there is no way to save it – so instead create your own web page that you can save on your hard drive.  For example, you can create “Temp1,htm”, using a text editor (Notepad).  Your web page really only needs a single line in it – the link.  Save the file as “Temp1.htm” – replace the link below with your own.  I provide an actual working example that you can try, after this (see below) :<a href=http://www.sample.com/sample.mov&#8221;>Link</a>
  • open Temp1.htm in your browser, right-click on your link, and select “Save Target as . . .”
  • Done

An actual working sample of the “Create your own Web page” trick:

OK here is a link to an old, grainy video of my daughter:  My daughter  –  I used a “ram” file, which is nothing more than a text file with a link to the actual clip, which is an rm file.

Test the file first, and play it, as you would on any website.  Now suppose you want to save the actual clip, which is an rm file.  You don’t know where the file resides, since the link does not go directly to that file.  So do the following:

right-click on the link and select “Save Target as. . .” and then save the ram file

open the ram file (daughter.ram) in notepad – and you will see it has one line:

http://www.infocellar.com/funstuff/Videos/daughter2.rm

now, you can enter that line in your web browser’s address field and play the file – but you still cannot save it.  So instead, create a web page from notepad, and save it as “c:\temp1.htm“.  The web page will have the following text in it:

<a href=http://www.infocellar.com/funstuff/Videos/daughter2.rm&#8221;>Link</a>

now just run the file, temp1.htm and it will open in your web browser and will have a single link in it to the rm file.  Roght-click and select “Save Target as” to get the rm file.  Give it any name you wish, such as “temp1.rm”.  Oddly, IE will still tell you it is downloading a “ram” file, but it will actually be downloading the rm.

Saving embedded video Flash (swf files)

Download the Free version of Flash Extract Pilot.

*** see http://www.flash-mx-template.com/ to buy cool custom Flash 5.0 templates that you can customize (you get the source code – the *.fla file) – or have them make a special template for you !!  Of course, you need Flash MX to edit the fla file.

Easy to save – manually or using a flash capture utility.  These are movies by Macromedia software – used for web page intro’s normally.  The entire movie, which is an “swf” file, is saved before it begins playing.  Simply search the “Temporary Internet Files” folder for *.swf, and then copy the file somewhere.

NOTE: to se if your Flash and Shockwave players are up to date, go to the Macromedia Test Page.

Flash Capture, by Dreamingsoft

This is an excellent flash capture utility.  Once installed, for any flash movie that is playing, you just move your mouse over it and a tiny box pops up with an option to save the swf file wherever you choose !!  They have a free 14-day trial, but it does not say the trial is disabled after that time – seems to just give a nag screen. There are several other similar utilities and as mentioned – Flash Movie Extract Pilot is a freebie.

How to embed your saved SWF File Into a Web Page

Just follow this sample code, which works for the Flash Movie file, M03L03P002.swf  –  simply replace that file name with your own, and adjust the display size (width and height) parameters, which appear in two places:

<body>
http://M03L03P002.swf
</body>
</html>

Background and the Difference between Flash and Shockwave

*** see also Flash vs Shockwave 

Shockwave was created by Macromedia, to enable webbers to view Macromedia Director content directly in their browser. Macromedia also acquired and quickly updated and renamed the program to Flash 2, and the plug-in to Shockwave Flash Player. That, was when all the confusion began.  Basically there are two players:

  • Shockwave Director Player, plays Director files.
  • Shockwave Flash Player, plays Flash files.

Shockwave usually refers to content created in Director and is used for interactive web add-ins, especially games.  Flash refers to content created in Flash and is used mostly for movies, such as web page intros.

Saving streaming Flash Videos

FLV files can be delivered in three different ways:

  • Embedded in a SWF file using the Flash authoring tool (supported in Flash Player 6 and later)
  • Streamed via RTMP to the Flash Player using the Flash Media Server (formerly called Flash Communication Server)
    This is the best option for allowing advanced seek capabilities and automatic bandwidth detection.
  • Progressively downloaded via HTTP (supported in Flash Player 7 and later.)

 

Saving Microsoft’s WMA, WMV, ASX, ASF, etc

Easy to save manually.  These are Windows Media Audio, Windows Media Video, Active Stream redirector and Advanced Systems Format.  These are true streaming media files, in that they begin playing as soon as they have buffered (saved) enough data so that the clip can play without stopping in the middle.  Of course, the algorithm that calculates how much to buffer before playing – makes a guess, which is often wrong, and the clip may stop several times to re-buffer.  The point is, the file is not complete until you have played the entire clip!!!

The following method works for wmv, wma, asf, and asx files :

  1. from the website, start playing the stream – if it ask you if you want to play it in IE, say NO !!
  2. let the clip finish – DO NOT close WMplayer and DO NOT go to another web page
  3. open Explorer and go to the Temporary Internet Files folder
  4. now find the stored file . . . either :
    – search for files such as *.wmv, *.wma, etc
    – click on the “Date Modified” tab which will be to the right of the Search box headers (you may have to
    scroll to the right to see it). Then look for the most recent files.
  5. copy the file to another folder and rename it if necessary – DO NOT PLAY IT IN THE “Temporary Internet Files” folder
  6. DONE

ASF and ASX

ASF is a streaming video or audio file – similar to Real Media rm files.  It has extra capabilities such as slide shows and special controls.

ASX comes in 2 flavors.  It was designed to be a Windows Metafile Extension, which is a text file that makes a call to the ASF file, similar to Real Media ram files  –  but it can also be a self-contained video or audio file.   

  • ASX text file –  a Stream Redirector text file based on eXtensible Markup Language (XML), and contains the URL to a streaming media ASF file.  
  • ASX stand-alone file  –  contain the actual video or audio.

 

Saving RA, RM (Real Audio and Real Media  –  ra is audio and rm is video) 

Usually Impossible***  to save manually  –  requires a Real Media capture utility.  These are also true streaming media files – unfortunately, only the pointer file is downloaded to Temporary Internet Files, and the actual media file is either held in mempry or stored with another filename.

*** in some cases, the website author will link directly to the ra or rm file instead of using a ram pointer file.  In these cases you can use the same method as listed above for Microsoft streaming media files – they will be stored in the “Temporary Internet Files” folder.

RAM and SMI files – Real Media (*.rm and *.ra) files are not accessed directly by the link.  For example, to play mysong.ra, the web page must include a ram file – that is a text reference (pointer), for example, mysong.ram.  The ram file has one short line of text, with the URL of the Real Media clip.  The web page will have a link to the ram file only.

Realmedia has gotten smart lately, and created SMI files to replace the older, limited RAM files. The smi file is an in-between link which rerouts the link yet again (therefore utilities such as xfileget, streamdpwn and streambox vcr only downloads the .smi file and gets fooled into thinking that’s the whole thing).  “SMI” files are similar to RAM files, but they have a lot of additional features and parameters that can be added.  Again, the SMI files merely points to the Real Media files, so you cannot download it from the link.

The line of text in the ram file begins with the protocol used to initiate streaming and can be rtsp://, mms://, pnm://, or http://  -  however, http:// and pnm:// are the most common.  But Do not use pnm:// in the playback URL for a live or on-demand stream generated with Helix Producer 9. Files generated with Helix Producer 9 or later do not include the interleaved audio/video stream required by the PNM protocol.  

NOTE:  the pnm prefix  -  if the ram file text line begins with  "pnm://"  instead of "http://" then the file resides on a “Real Audio server”. 

see Intro to Streaming Media by Real Networks - a MUST READ !!!

and  http://service.real.com/help/library/encoders.html for tech info

StreamDown utility (saves those “hard to get” Real Media files)

You can either buy ($39) or download the 15-day Trial of “CoCSoft Stream Down“.  The trial is 15 days but see the modified date trick below.  Or if that does not work, re-download the trial.  It supports HTTP, FTP, RTSP, MMS, MMSU, and MMST.  Simply paste the URL, and it downloads the associated media file !!
Minor Bug – saved extension wrong:  as an example, if you paste the URL, http://www.acme.com/file1.ram  –  for some reason, StreamDown saves the ra file but it names it with a ram extension – so you need to rename it with an ra extension.
Modified Date trick – to change the modified date of a file, open it in any Hex editor such as XVI32, change one Hex value, save it, change it back, and save it again – viola – new modified date !!  This usually does not work with trials, however – they have protection built in to defeat that. But with StreamDown it gets confused and in one case, said the user had -8 days left, and kept working !!!

X-FileGet Utility

RAM only – DOES NOT WORK WITH SMI files !!!  Similar to StreamDown – I tested this app and it did not work !!

Why the Manual Method of saving Real Media Files does not Work

Web browsers will open a URL with a ram file at the end and play the ra file in the Real Player – but they will not open a “page” with an ra or rm extension !!!

Now, it seems that if we could see the contents of the ram file we would see the URL to the ra file – but how to download it ??  Some say you can go to your browser’s file associations and remove the ra association – then your browser does not know how to play the file, and will ask you if you want to save it – but no, the browser will simply refuse to open that “page”.  In addition, you typically cannot FTP files from a URL because the hosts do not permit FTP in general, except for specific special FTP locations.
Google Videos (gvp files)

The “gvp” file is NOT the video !!!  It is merely a text file with the URL to the video inside of it.  It also has other text to fool you – but here we show you what text to use for the URL.  SIMPLE !!!

NOTE: “www.keepvids.com” used to be popular for saving Google videos – but Google changed something and keepvids it no longers works reliably – if at all !!!

  1. associate “gvp” files with Notepad as follows:
  • in Notepad, make a dummy text file with any words in it
  • save the text file as “temp.gvp” and close Notepad
  • run the file temp.gvp – click “Select the program from a list”
  • select “Notepad” as the Program to open the file in and
  1. check the box “Always use the selected program to open this kind of file”
  2. open two instances of IE web browser
  3. goto Google Videos (video.google.com) – or goto http://video.google.com/videoranking for the Top 100
  4. select a video and watch just enough of it to get it into your cache (until the grey bar goes all the way to the right)
  5. click the Stop button
  6. click “Download Video”
  7. click “Cancel” when it asks you to DL the Google Player (GooglePlayerSetup.exe)
  8. click “Manually download the video”
  9. click “Open” – this causes a small text file with the extension *.gvp” to open in Notepad
  10. drag mouse across the text that is between “url:” and “duration” (see example below)
  11. hit CTRL-C to copy the URL of the video clip and close Notepad
  12. goto your other Web Browser (not the one running Google Video)
  13. click once in the Address field to get the cursor there
  14. hit CTRL-V to paste the video URL, then <enter>
  15. a box pops up – click the “Save” button

DONE !!!  The process looks cumbersome, but once you do it a couple of times it takes all of 10 seconds !!

You can now go back to the other IE window that is running Google Videos, and click the “Back” button to view and save more videos

Example of a Google gvp file and the actual URL of the Video

The gvp file contains the following text:

# download the free Google Video Player from http://video.google.com/
gvp_version:1.1
url:http://vp.video.google.com/videodownload?version=0&secureurl=uQAAAPzDOYe-vYLD13Nz9iShbRiqlOGj1ZnKPz6bUjhnLTT8k2J4f1CN86aS2P17lp3e4uFpsnR3jvc-qGvW5wH6tHEHPPqJXFEv1CESF7ROn0G5Wg802X4LougBLCrYhp4mgErkHaidHVYlgv3tQfZB5InyA1hvU5jhqM4Dyzt-2RwEG7I7UPpdjiKhrBbwwzZ8-3eMoHUUHf1ZiTejCcoUqUHYgOJySzPqvJkOtCl4tiTsAiKAjhoICiFxZuLZ_o5quA&sigh=GfxvGl1Xtuj4X95WxnOwqsCmjxY&begin=0&len=10109&docid=-5737652982419375770
docid:-5737652982419375770
duration:10109
title:Teen girls fight in school hallway. Stock Footage.
description:Medium shot of two teen girls standing and fighting in hallway at school. Thinkstock Footage is live on Creatas. To purchase this clip please click the link above.

Here I have inserted dashes and arrows to show you where the text that you want begins and ends:

# download the free Google Video Player from http://video.google.com/
gvp_version:1.1
url:———–>http://vp.video.google.com/videodownload?version=0&secureurl=uQAAAPzDOYe-vYLD13Nz9iShbRiqlOGj1ZnKPz6bUjhnLTT8k2J4f1CN86aS2P17lp3e4uFpsnR3jvc-qGvW5wH6tHEHPPqJXFEv1CESF7ROn0G5Wg802X4LougBLCrYhp4mgErkHaidHVYlgv3tQfZB5InyA1hvU5jhqM4Dyzt-2RwEG7I7UPpdjiKhrBbwwzZ8-3eMoHUUHf1ZiTejCcoUqUHYgOJySzPqvJkOtCl4tiTsAiKAjhoICiFxZuLZ_o5quA&sigh=GfxvGl1Xtuj4X95WxnOwqsCmjxY&begin=0&len=10109&docid=-5737652982419375770
docid:-5737652982419375770<———–
duration:10109
title:Teen girls fight in school hallway. Stock Footage.
description:Medium shot of two teen girls standing and fighting in hallway at school. Thinkstock Footage is live on Creatas. To purchase this clip please click the link above.

So when you select that text, it will be the following string:

http://vp.video.google.com/videodownload?version=0&secureurl=uQAAAPzDOYe-vYLD13Nz9iShbRiqlOGj1ZnKPz6bUjhnLTT8k2J4f1CN86aS2P17lp3e4uFpsnR3jvc-qGvW5wH6tHEHPPqJXFEv1CESF7ROn0G5Wg802X4LougBLCrYhp4mgErkHaidHVYlgv3tQfZB5InyA1hvU5jhqM4Dyzt-2RwEG7I7UPpdjiKhrBbwwzZ8-3eMoHUUHf1ZiTejCcoUqUHYgOJySzPqvJkOtCl4tiTsAiKAjhoICiFxZuLZ_o5quA&sigh=GfxvGl1Xtuj4X95WxnOwqsCmjxY&begin=0&len=10109&docid=-5737652982419375770
docid:-5737652982419375770

 

Essbase Shell How To

How To Use ESSBASE SHELL Command

 

By W.ZH

 

1. Start Shell

For windows, essmsh.exe in the C:\Hyperion\AnalyticServices\bin or in the admin folder C:\Hyperion\AnalyticAdministrationServices\server\bin

essmsh -h | more              to get help

essmsh -l user password       to login

essmsh filename               to use a file as input (can login in file)

            Note: this user name is not the admin account for administration service, this account is account to start the essbase server, it is the essbase ‘s registered username

login username password on localhost;

 

5. spool on to ‘C:\esslog.log’;                 to log the input and output into a logfile, can use ‘\’

spool stdout on to ‘output.txt’;

spool stderr on to ‘errors.txt’;

 

6. spool off                  to stop the log

 

7. echo $ARBORPATH            to display the path setting of essbase server

 

8. version;             to see version of MaxL

 

9. logout;              to loglout

 

10.exit;                exit the shell

 

11. set var1 = sample;        to set local variable

 

12. echo $var1;         to see what the value of $var1 is

 

13. /* comments */            to comment

 

14. using $?                  to display last command’s return, 0 is ok, 1 is error in exec

 

Start Essbase in Windows

How to start Essbase service in windows server

By W.ZH

 

0. ensure your DB for Essbase is running, it could be any DB(Oracle, mySQL, MSSQL, etc.) when you set up in you installation of essbase.

 

1. start the essbase server and sample in Hyperion–>essbase, you need input the server password to start it.

 

2. start the administration service server in Hyperion–>administration service

 

3. start the administration service console in Hyperion–>administration service, input your admin account and password.

 

4. then can start use it and monitor it.